2024-04-26T00:52:11.360 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T00:52:11.360 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi --verbose --owner scheduled_teuthology@teuthology --description rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} --archive /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968 -- /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/orig.config.yaml 2024-04-26T00:52:11.405 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T00:52:11.497 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968 branch: squid description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} email: ceph-qa@ceph.com first_in_suite: false job_id: '7673968' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.1 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 9886 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 30270/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a targets: smithi008.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK4frcFiXxuvFihce9UuoZoisKkhLyfeOBzqyu9QD0Ag0Ib9TqpDIqR2y/Kw1qisy2aNXGlAAJnadpePsnNBvdA= smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJv4HnRZVX1vvurODkhJ8tjR5Yt3zLZhRKjDfe5yPtS6hL4vO6RwTBiRjDbxfjdXiolat12eFeWP8Z74UFf2nC4= smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJspkZF0hdTqWWhaEn4cSjWa0GPSlwvhE+s0JRCJAovLtBTD51U4AfMgOb3dloOtVSQDLdhnXBKP6bI3g/jf4yQ= smithi164.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNQJPJnLOnvuCUj2BVAR4HkQSosOA0k3Q9YV8c3hcP5RMq297jKmS9Ae/e/iHxYNT0dpkdDmJXfeHeIE+YweZGo= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: bluestore_new_sharding: random chance_bluestore_reshard: 1 chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_21:00:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T00:52:11.497 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa; will attempt to use it 2024-04-26T00:52:11.498 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks 2024-04-26T00:52:11.498 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T00:52:11.500 INFO:teuthology.task.internal:Checking packages... 2024-04-26T00:52:11.533 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-26T00:52:11.533 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T00:52:11.533 INFO:teuthology.packaging:ref: None 2024-04-26T00:52:11.533 INFO:teuthology.packaging:tag: None 2024-04-26T00:52:11.533 INFO:teuthology.packaging:branch: squid 2024-04-26T00:52:11.533 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T00:52:11.534 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-04-26T00:52:11.757 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2447-gb22e2ebd-1jammy 2024-04-26T00:52:11.758 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T00:52:11.765 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T00:52:11.765 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T00:52:11.818 INFO:teuthology.task.internal:Saving configuration 2024-04-26T00:52:11.834 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T00:52:11.843 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T00:52:11.868 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi008.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 00:46:13.406502', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC5d6MNpzUxNjyGHRF/jJDpJXGR9h9MjAIEk3iAQUUzK'} 2024-04-26T00:52:11.888 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 00:46:13.404586', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEyNFdnwD81DSqVw6K408edXBfFTfRcji0pWOtBccDYx'} 2024-04-26T00:52:11.907 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 00:46:13.408157', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO10tvssGfoj0QeaWAXyh3YNW2oyq51rqZNyrSHYizD9'} 2024-04-26T00:52:11.926 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 00:46:13.409854', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEDsmrcks+uVSY8zODYAZYYL37hfCw5QKE4f7C4lDdS7'} 2024-04-26T00:52:11.927 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T00:52:11.934 INFO:teuthology.task.internal:roles: ubuntu@smithi008.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-04-26T00:52:11.934 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-04-26T00:52:11.934 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-04-26T00:52:11.934 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-04-26T00:52:11.934 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T00:52:12.085 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f32be44b280>, signals=[15]) 2024-04-26T00:52:12.086 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T00:52:12.092 INFO:teuthology.task.internal:Opening connections... 2024-04-26T00:52:12.092 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi008.front.sepia.ceph.com 2024-04-26T00:52:12.094 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:52:12.181 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2024-04-26T00:52:12.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:52:12.263 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-04-26T00:52:12.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:52:12.342 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2024-04-26T00:52:12.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:52:12.419 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T00:52:12.426 DEBUG:teuthology.orchestra.run.smithi008:> uname -m 2024-04-26T00:52:12.432 INFO:teuthology.orchestra.run.smithi008.stdout:x86_64 2024-04-26T00:52:12.432 DEBUG:teuthology.orchestra.run.smithi008:> cat /etc/os-release 2024-04-26T00:52:12.480 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T00:52:12.481 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="Ubuntu" 2024-04-26T00:52:12.481 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="22.04" 2024-04-26T00:52:12.481 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T00:52:12.481 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_CODENAME=jammy 2024-04-26T00:52:12.481 INFO:teuthology.orchestra.run.smithi008.stdout:ID=ubuntu 2024-04-26T00:52:12.481 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE=debian 2024-04-26T00:52:12.481 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T00:52:12.481 INFO:teuthology.orchestra.run.smithi008.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T00:52:12.481 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T00:52:12.481 INFO:teuthology.orchestra.run.smithi008.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T00:52:12.481 INFO:teuthology.orchestra.run.smithi008.stdout:UBUNTU_CODENAME=jammy 2024-04-26T00:52:12.482 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2024-04-26T00:52:12.502 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2024-04-26T00:52:12.508 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2024-04-26T00:52:12.508 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2024-04-26T00:52:12.554 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T00:52:12.554 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="Ubuntu" 2024-04-26T00:52:12.554 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="22.04" 2024-04-26T00:52:12.554 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T00:52:12.554 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_CODENAME=jammy 2024-04-26T00:52:12.555 INFO:teuthology.orchestra.run.smithi012.stdout:ID=ubuntu 2024-04-26T00:52:12.555 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE=debian 2024-04-26T00:52:12.555 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T00:52:12.555 INFO:teuthology.orchestra.run.smithi012.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T00:52:12.555 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T00:52:12.555 INFO:teuthology.orchestra.run.smithi012.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T00:52:12.555 INFO:teuthology.orchestra.run.smithi012.stdout:UBUNTU_CODENAME=jammy 2024-04-26T00:52:12.555 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2024-04-26T00:52:12.576 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-04-26T00:52:12.582 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-04-26T00:52:12.583 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-04-26T00:52:12.630 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T00:52:12.630 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="Ubuntu" 2024-04-26T00:52:12.630 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="22.04" 2024-04-26T00:52:12.631 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T00:52:12.631 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_CODENAME=jammy 2024-04-26T00:52:12.631 INFO:teuthology.orchestra.run.smithi116.stdout:ID=ubuntu 2024-04-26T00:52:12.631 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE=debian 2024-04-26T00:52:12.631 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T00:52:12.631 INFO:teuthology.orchestra.run.smithi116.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T00:52:12.631 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T00:52:12.632 INFO:teuthology.orchestra.run.smithi116.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T00:52:12.632 INFO:teuthology.orchestra.run.smithi116.stdout:UBUNTU_CODENAME=jammy 2024-04-26T00:52:12.632 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-04-26T00:52:12.651 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2024-04-26T00:52:12.657 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2024-04-26T00:52:12.658 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2024-04-26T00:52:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T00:52:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="Ubuntu" 2024-04-26T00:52:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="22.04" 2024-04-26T00:52:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T00:52:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_CODENAME=jammy 2024-04-26T00:52:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:ID=ubuntu 2024-04-26T00:52:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE=debian 2024-04-26T00:52:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T00:52:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T00:52:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T00:52:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T00:52:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:UBUNTU_CODENAME=jammy 2024-04-26T00:52:12.705 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2024-04-26T00:52:12.726 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T00:52:12.738 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T00:52:12.760 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T00:52:12.760 DEBUG:teuthology.orchestra.run.smithi008:> test '!' -e /home/ubuntu/cephtest 2024-04-26T00:52:12.763 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2024-04-26T00:52:12.765 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-04-26T00:52:12.768 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2024-04-26T00:52:12.772 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T00:52:12.785 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T00:52:12.785 DEBUG:teuthology.orchestra.run.smithi008:> test -z $(ls -A /var/lib/ceph) 2024-04-26T00:52:12.810 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2024-04-26T00:52:12.812 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-04-26T00:52:12.815 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2024-04-26T00:52:12.832 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T00:52:12.930 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T00:52:12.957 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-26T00:52:12.957 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-26T00:52:12.957 DEBUG:teuthology.orchestra.run.smithi008:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T00:52:12.958 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T00:52:12.958 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T00:52:12.959 DEBUG:teuthology.orchestra.run.smithi164:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T00:52:12.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T00:52:12.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T00:52:12.964 DEBUG:teuthology.orchestra.run.smithi008:> uname -r 2024-04-26T00:52:12.965 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-04-26T00:52:12.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T00:52:12.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T00:52:12.967 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-04-26T00:52:12.967 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-04-26T00:52:13.012 INFO:teuthology.orchestra.run.smithi116.stdout:5.15.0-56-generic 2024-04-26T00:52:13.012 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.15.0-56-generic 2024-04-26T00:52:13.012 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get clean 2024-04-26T00:52:13.014 INFO:teuthology.orchestra.run.smithi008.stdout:5.15.0-56-generic 2024-04-26T00:52:13.014 INFO:teuthology.orchestra.run.smithi164.stdout:5.15.0-56-generic 2024-04-26T00:52:13.015 INFO:teuthology.task.kernel:Running kernel on smithi008: 5.15.0-56-generic 2024-04-26T00:52:13.015 DEBUG:teuthology.orchestra.run.smithi008:> sudo apt-get clean 2024-04-26T00:52:13.015 INFO:teuthology.task.kernel:Running kernel on smithi164: 5.15.0-56-generic 2024-04-26T00:52:13.015 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-get clean 2024-04-26T00:52:13.016 INFO:teuthology.orchestra.run.smithi012.stdout:5.15.0-56-generic 2024-04-26T00:52:13.016 INFO:teuthology.task.kernel:Running kernel on smithi012: 5.15.0-56-generic 2024-04-26T00:52:13.017 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get clean 2024-04-26T00:52:13.128 DEBUG:teuthology.orchestra.run.smithi008:> sudo apt-get update 2024-04-26T00:52:13.138 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get update 2024-04-26T00:52:13.144 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-get update 2024-04-26T00:52:13.145 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get update 2024-04-26T00:52:13.280 INFO:teuthology.orchestra.run.smithi012.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T00:52:13.302 INFO:teuthology.orchestra.run.smithi164.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T00:52:13.392 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T00:52:13.406 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T00:52:13.416 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T00:52:13.430 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T00:52:13.430 INFO:teuthology.orchestra.run.smithi008.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T00:52:13.474 INFO:teuthology.orchestra.run.smithi012.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T00:52:13.485 INFO:teuthology.orchestra.run.smithi008.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T00:52:13.491 INFO:teuthology.orchestra.run.smithi164.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T00:52:13.511 INFO:teuthology.orchestra.run.smithi116.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T00:52:13.511 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T00:52:13.513 INFO:teuthology.orchestra.run.smithi012.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T00:52:13.513 INFO:teuthology.orchestra.run.smithi164.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T00:52:13.642 INFO:teuthology.orchestra.run.smithi164.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T00:52:13.642 INFO:teuthology.orchestra.run.smithi012.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T00:52:13.812 INFO:teuthology.orchestra.run.smithi012.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T00:52:13.819 INFO:teuthology.orchestra.run.smithi164.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T00:52:13.821 INFO:teuthology.orchestra.run.smithi164.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T00:52:13.821 INFO:teuthology.orchestra.run.smithi012.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T00:52:13.821 INFO:teuthology.orchestra.run.smithi012.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T00:52:13.822 INFO:teuthology.orchestra.run.smithi012.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T00:52:13.822 INFO:teuthology.orchestra.run.smithi164.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T00:52:13.822 INFO:teuthology.orchestra.run.smithi164.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T00:52:13.836 INFO:teuthology.orchestra.run.smithi116.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T00:52:13.839 INFO:teuthology.orchestra.run.smithi012.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T00:52:13.845 INFO:teuthology.orchestra.run.smithi008.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T00:52:13.845 INFO:teuthology.orchestra.run.smithi008.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T00:52:13.877 INFO:teuthology.orchestra.run.smithi164.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T00:52:13.885 INFO:teuthology.orchestra.run.smithi012.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T00:52:13.888 INFO:teuthology.orchestra.run.smithi164.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T00:52:13.897 INFO:teuthology.orchestra.run.smithi164.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T00:52:13.897 INFO:teuthology.orchestra.run.smithi164.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T00:52:13.903 INFO:teuthology.orchestra.run.smithi012.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T00:52:13.903 INFO:teuthology.orchestra.run.smithi012.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T00:52:13.920 INFO:teuthology.orchestra.run.smithi012.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T00:52:13.922 INFO:teuthology.orchestra.run.smithi164.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T00:52:13.951 INFO:teuthology.orchestra.run.smithi164.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T00:52:13.957 INFO:teuthology.orchestra.run.smithi164.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T00:52:13.957 INFO:teuthology.orchestra.run.smithi164.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T00:52:13.967 INFO:teuthology.orchestra.run.smithi012.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T00:52:13.969 INFO:teuthology.orchestra.run.smithi012.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T00:52:13.969 INFO:teuthology.orchestra.run.smithi012.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T00:52:13.969 INFO:teuthology.orchestra.run.smithi012.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T00:52:13.969 INFO:teuthology.orchestra.run.smithi012.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T00:52:13.969 INFO:teuthology.orchestra.run.smithi012.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T00:52:13.974 INFO:teuthology.orchestra.run.smithi116.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T00:52:13.977 INFO:teuthology.orchestra.run.smithi012.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T00:52:13.978 INFO:teuthology.orchestra.run.smithi008.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T00:52:13.980 INFO:teuthology.orchestra.run.smithi164.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T00:52:13.980 INFO:teuthology.orchestra.run.smithi164.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T00:52:13.980 INFO:teuthology.orchestra.run.smithi164.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T00:52:14.074 INFO:teuthology.orchestra.run.smithi164.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T00:52:14.127 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T00:52:14.169 INFO:teuthology.orchestra.run.smithi008.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T00:52:14.177 INFO:teuthology.orchestra.run.smithi012.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T00:52:14.178 INFO:teuthology.orchestra.run.smithi008.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T00:52:14.299 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T00:52:14.324 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T00:52:14.327 INFO:teuthology.orchestra.run.smithi012.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T00:52:14.329 INFO:teuthology.orchestra.run.smithi008.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T00:52:14.333 INFO:teuthology.orchestra.run.smithi008.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T00:52:14.339 INFO:teuthology.orchestra.run.smithi116.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T00:52:14.340 INFO:teuthology.orchestra.run.smithi116.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T00:52:14.341 INFO:teuthology.orchestra.run.smithi012.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T00:52:14.342 INFO:teuthology.orchestra.run.smithi012.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T00:52:14.344 INFO:teuthology.orchestra.run.smithi012.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T00:52:14.344 INFO:teuthology.orchestra.run.smithi008.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T00:52:14.345 INFO:teuthology.orchestra.run.smithi008.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T00:52:14.346 INFO:teuthology.orchestra.run.smithi008.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T00:52:14.347 INFO:teuthology.orchestra.run.smithi008.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T00:52:14.347 INFO:teuthology.orchestra.run.smithi008.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T00:52:14.402 INFO:teuthology.orchestra.run.smithi116.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T00:52:14.403 INFO:teuthology.orchestra.run.smithi116.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T00:52:14.406 INFO:teuthology.orchestra.run.smithi012.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T00:52:14.408 INFO:teuthology.orchestra.run.smithi008.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T00:52:14.410 INFO:teuthology.orchestra.run.smithi116.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T00:52:14.413 INFO:teuthology.orchestra.run.smithi012.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T00:52:14.416 INFO:teuthology.orchestra.run.smithi008.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T00:52:14.429 INFO:teuthology.orchestra.run.smithi008.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T00:52:14.452 INFO:teuthology.orchestra.run.smithi008.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T00:52:14.458 INFO:teuthology.orchestra.run.smithi008.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T00:52:14.458 INFO:teuthology.orchestra.run.smithi008.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T00:52:14.470 INFO:teuthology.orchestra.run.smithi008.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T00:52:14.575 INFO:teuthology.orchestra.run.smithi116.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T00:52:14.577 INFO:teuthology.orchestra.run.smithi012.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T00:52:14.582 INFO:teuthology.orchestra.run.smithi008.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T00:52:14.649 INFO:teuthology.orchestra.run.smithi008.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T00:52:15.055 INFO:teuthology.orchestra.run.smithi008.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T00:52:15.061 INFO:teuthology.orchestra.run.smithi008.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T00:52:15.075 INFO:teuthology.orchestra.run.smithi008.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T00:52:15.075 INFO:teuthology.orchestra.run.smithi008.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T00:52:15.078 INFO:teuthology.orchestra.run.smithi008.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T00:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T00:52:15.161 INFO:teuthology.orchestra.run.smithi012.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T00:52:15.174 INFO:teuthology.orchestra.run.smithi008.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T00:52:15.197 INFO:teuthology.orchestra.run.smithi164.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T00:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T00:52:15.272 INFO:teuthology.orchestra.run.smithi012.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T00:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T00:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T00:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T00:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T00:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T00:52:15.288 INFO:teuthology.orchestra.run.smithi008.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T00:52:15.288 INFO:teuthology.orchestra.run.smithi116.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T00:52:15.292 INFO:teuthology.orchestra.run.smithi116.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T00:52:15.292 INFO:teuthology.orchestra.run.smithi012.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T00:52:15.294 INFO:teuthology.orchestra.run.smithi012.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T00:52:15.296 INFO:teuthology.orchestra.run.smithi116.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T00:52:15.298 INFO:teuthology.orchestra.run.smithi012.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T00:52:15.298 INFO:teuthology.orchestra.run.smithi012.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T00:52:15.299 INFO:teuthology.orchestra.run.smithi012.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T00:52:15.299 INFO:teuthology.orchestra.run.smithi012.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T00:52:15.303 INFO:teuthology.orchestra.run.smithi012.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T00:52:15.307 INFO:teuthology.orchestra.run.smithi012.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T00:52:15.308 INFO:teuthology.orchestra.run.smithi008.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T00:52:15.309 INFO:teuthology.orchestra.run.smithi008.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T00:52:15.309 INFO:teuthology.orchestra.run.smithi008.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T00:52:15.313 INFO:teuthology.orchestra.run.smithi008.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T00:52:15.313 INFO:teuthology.orchestra.run.smithi008.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T00:52:15.314 INFO:teuthology.orchestra.run.smithi008.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T00:52:15.318 INFO:teuthology.orchestra.run.smithi008.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T00:52:15.323 INFO:teuthology.orchestra.run.smithi008.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T00:52:15.351 INFO:teuthology.orchestra.run.smithi116.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T00:52:15.461 INFO:teuthology.orchestra.run.smithi116.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T00:52:15.472 INFO:teuthology.orchestra.run.smithi012.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T00:52:15.488 INFO:teuthology.orchestra.run.smithi008.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T00:52:15.543 INFO:teuthology.orchestra.run.smithi116.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T00:52:15.554 INFO:teuthology.orchestra.run.smithi012.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T00:52:15.571 INFO:teuthology.orchestra.run.smithi008.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T00:52:15.635 INFO:teuthology.orchestra.run.smithi012.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T00:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T00:52:15.689 INFO:teuthology.orchestra.run.smithi012.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T00:52:15.692 INFO:teuthology.orchestra.run.smithi008.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T00:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T00:52:15.736 INFO:teuthology.orchestra.run.smithi116.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T00:52:15.737 INFO:teuthology.orchestra.run.smithi008.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T00:52:15.741 INFO:teuthology.orchestra.run.smithi012.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T00:52:15.772 INFO:teuthology.orchestra.run.smithi008.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T00:52:17.594 INFO:teuthology.orchestra.run.smithi116.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T00:52:17.902 INFO:teuthology.orchestra.run.smithi116.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T00:52:17.926 INFO:teuthology.orchestra.run.smithi116.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T00:52:18.076 INFO:teuthology.orchestra.run.smithi164.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T00:52:18.422 INFO:teuthology.orchestra.run.smithi164.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T00:52:18.463 INFO:teuthology.orchestra.run.smithi164.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T00:52:18.581 INFO:teuthology.orchestra.run.smithi164.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T00:52:20.262 INFO:teuthology.orchestra.run.smithi012.stdout:Fetched 13.2 MB in 3s (5,230 kB/s) 2024-04-26T00:52:20.509 INFO:teuthology.orchestra.run.smithi008.stdout:Fetched 13.2 MB in 3s (5,174 kB/s) 2024-04-26T00:52:20.787 INFO:teuthology.orchestra.run.smithi116.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T00:52:20.940 INFO:teuthology.orchestra.run.smithi116.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T00:52:21.656 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-04-26T00:52:21.657 INFO:teuthology.orchestra.run.smithi116.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T00:52:21.669 INFO:teuthology.orchestra.run.smithi116.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T00:52:21.672 DEBUG:teuthology.orchestra.run.smithi012:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T00:52:21.727 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-04-26T00:52:21.844 INFO:teuthology.orchestra.run.smithi008.stdout:Reading package lists... 2024-04-26T00:52:21.864 DEBUG:teuthology.orchestra.run.smithi008:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T00:52:21.908 INFO:teuthology.orchestra.run.smithi012.stdout:Building dependency tree... 2024-04-26T00:52:21.908 INFO:teuthology.orchestra.run.smithi012.stdout:Reading state information... 2024-04-26T00:52:21.934 INFO:teuthology.orchestra.run.smithi008.stdout:Reading package lists... 2024-04-26T00:52:22.071 INFO:teuthology.orchestra.run.smithi012.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T00:52:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T00:52:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T00:52:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T00:52:22.072 INFO:teuthology.orchestra.run.smithi012.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T00:52:22.072 INFO:teuthology.orchestra.run.smithi012.stdout: python2.7-minimal 2024-04-26T00:52:22.073 INFO:teuthology.orchestra.run.smithi012.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T00:52:22.073 INFO:teuthology.orchestra.run.smithi012.stdout:The following additional packages will be installed: 2024-04-26T00:52:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T00:52:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T00:52:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T00:52:22.074 INFO:teuthology.orchestra.run.smithi012.stdout:Suggested packages: 2024-04-26T00:52:22.074 INFO:teuthology.orchestra.run.smithi012.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T00:52:22.074 INFO:teuthology.orchestra.run.smithi012.stdout:Recommended packages: 2024-04-26T00:52:22.074 INFO:teuthology.orchestra.run.smithi012.stdout: thermald 2024-04-26T00:52:22.127 INFO:teuthology.orchestra.run.smithi012.stdout:The following NEW packages will be installed: 2024-04-26T00:52:22.127 INFO:teuthology.orchestra.run.smithi012.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T00:52:22.127 INFO:teuthology.orchestra.run.smithi012.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T00:52:22.128 INFO:teuthology.orchestra.run.smithi012.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T00:52:22.128 INFO:teuthology.orchestra.run.smithi012.stdout:The following packages will be upgraded: 2024-04-26T00:52:22.129 INFO:teuthology.orchestra.run.smithi012.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T00:52:22.134 INFO:teuthology.orchestra.run.smithi008.stdout:Building dependency tree... 2024-04-26T00:52:22.135 INFO:teuthology.orchestra.run.smithi008.stdout:Reading state information... 2024-04-26T00:52:22.281 INFO:teuthology.orchestra.run.smithi164.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T00:52:22.309 INFO:teuthology.orchestra.run.smithi012.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T00:52:22.309 INFO:teuthology.orchestra.run.smithi012.stdout:Need to get 113 MB of archives. 2024-04-26T00:52:22.309 INFO:teuthology.orchestra.run.smithi012.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T00:52:22.310 INFO:teuthology.orchestra.run.smithi012.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T00:52:22.317 INFO:teuthology.orchestra.run.smithi008.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T00:52:22.317 INFO:teuthology.orchestra.run.smithi008.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T00:52:22.317 INFO:teuthology.orchestra.run.smithi008.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T00:52:22.318 INFO:teuthology.orchestra.run.smithi008.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T00:52:22.318 INFO:teuthology.orchestra.run.smithi008.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T00:52:22.318 INFO:teuthology.orchestra.run.smithi008.stdout: python2.7-minimal 2024-04-26T00:52:22.318 INFO:teuthology.orchestra.run.smithi008.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T00:52:22.319 INFO:teuthology.orchestra.run.smithi008.stdout:The following additional packages will be installed: 2024-04-26T00:52:22.319 INFO:teuthology.orchestra.run.smithi008.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T00:52:22.319 INFO:teuthology.orchestra.run.smithi008.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T00:52:22.320 INFO:teuthology.orchestra.run.smithi008.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T00:52:22.321 INFO:teuthology.orchestra.run.smithi008.stdout:Suggested packages: 2024-04-26T00:52:22.321 INFO:teuthology.orchestra.run.smithi008.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T00:52:22.321 INFO:teuthology.orchestra.run.smithi008.stdout:Recommended packages: 2024-04-26T00:52:22.321 INFO:teuthology.orchestra.run.smithi008.stdout: thermald 2024-04-26T00:52:22.375 INFO:teuthology.orchestra.run.smithi008.stdout:The following NEW packages will be installed: 2024-04-26T00:52:22.375 INFO:teuthology.orchestra.run.smithi008.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T00:52:22.375 INFO:teuthology.orchestra.run.smithi008.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T00:52:22.375 INFO:teuthology.orchestra.run.smithi008.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T00:52:22.376 INFO:teuthology.orchestra.run.smithi008.stdout:The following packages will be upgraded: 2024-04-26T00:52:22.377 INFO:teuthology.orchestra.run.smithi008.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T00:52:22.556 INFO:teuthology.orchestra.run.smithi008.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T00:52:22.557 INFO:teuthology.orchestra.run.smithi008.stdout:Need to get 113 MB of archives. 2024-04-26T00:52:22.557 INFO:teuthology.orchestra.run.smithi008.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T00:52:22.557 INFO:teuthology.orchestra.run.smithi008.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T00:52:23.006 INFO:teuthology.orchestra.run.smithi116.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T00:52:23.214 INFO:teuthology.orchestra.run.smithi164.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T00:52:23.379 INFO:teuthology.orchestra.run.smithi164.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T00:52:23.721 INFO:teuthology.orchestra.run.smithi012.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T00:52:23.971 INFO:teuthology.orchestra.run.smithi164.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T00:52:23.983 INFO:teuthology.orchestra.run.smithi008.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T00:52:24.073 INFO:teuthology.orchestra.run.smithi012.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T00:52:24.084 INFO:teuthology.orchestra.run.smithi164.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T00:52:24.105 INFO:teuthology.orchestra.run.smithi164.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T00:52:24.106 INFO:teuthology.orchestra.run.smithi164.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T00:52:24.110 INFO:teuthology.orchestra.run.smithi164.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T00:52:24.110 INFO:teuthology.orchestra.run.smithi164.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T00:52:24.111 INFO:teuthology.orchestra.run.smithi164.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T00:52:24.111 INFO:teuthology.orchestra.run.smithi164.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T00:52:24.115 INFO:teuthology.orchestra.run.smithi164.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T00:52:24.120 INFO:teuthology.orchestra.run.smithi164.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T00:52:24.284 INFO:teuthology.orchestra.run.smithi164.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T00:52:24.321 INFO:teuthology.orchestra.run.smithi008.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T00:52:24.368 INFO:teuthology.orchestra.run.smithi164.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T00:52:24.449 INFO:teuthology.orchestra.run.smithi164.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T00:52:24.504 INFO:teuthology.orchestra.run.smithi164.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T00:52:24.555 INFO:teuthology.orchestra.run.smithi164.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T00:52:24.779 INFO:teuthology.orchestra.run.smithi116.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T00:52:25.334 INFO:teuthology.orchestra.run.smithi116.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T00:52:25.345 INFO:teuthology.orchestra.run.smithi116.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T00:52:25.372 INFO:teuthology.orchestra.run.smithi116.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T00:52:25.374 INFO:teuthology.orchestra.run.smithi116.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T00:52:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T00:52:26.348 INFO:teuthology.orchestra.run.smithi012.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T00:52:26.348 INFO:teuthology.orchestra.run.smithi012.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T00:52:26.349 INFO:teuthology.orchestra.run.smithi012.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T00:52:26.553 INFO:teuthology.orchestra.run.smithi008.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T00:52:26.553 INFO:teuthology.orchestra.run.smithi008.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T00:52:26.553 INFO:teuthology.orchestra.run.smithi008.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T00:52:26.700 INFO:teuthology.orchestra.run.smithi012.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T00:52:26.782 INFO:teuthology.orchestra.run.smithi012.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T00:52:27.185 INFO:teuthology.orchestra.run.smithi012.stdout:Fetched 113 MB in 5s (24.4 MB/s) 2024-04-26T00:52:27.352 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T00:52:27.393 INFO:teuthology.orchestra.run.smithi008.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T00:52:27.477 INFO:teuthology.orchestra.run.smithi008.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T00:52:27.901 INFO:teuthology.orchestra.run.smithi008.stdout:Fetched 113 MB in 5s (22.2 MB/s) 2024-04-26T00:52:28.059 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T00:52:28.921 INFO:teuthology.orchestra.run.smithi164.stdout:Fetched 13.2 MB in 11s (1,169 kB/s) 2024-04-26T00:52:29.363 INFO:teuthology.orchestra.run.smithi012.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T00:52:29.367 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:52:29.425 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:29.724 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 13.2 MB in 12s (1,089 kB/s) 2024-04-26T00:52:30.134 INFO:teuthology.orchestra.run.smithi008.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T00:52:30.138 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:52:30.248 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:30.379 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2024-04-26T00:52:30.397 DEBUG:teuthology.orchestra.run.smithi164:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T00:52:30.452 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2024-04-26T00:52:30.652 INFO:teuthology.orchestra.run.smithi164.stdout:Building dependency tree... 2024-04-26T00:52:30.653 INFO:teuthology.orchestra.run.smithi164.stdout:Reading state information... 2024-04-26T00:52:30.821 INFO:teuthology.orchestra.run.smithi164.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T00:52:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T00:52:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T00:52:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T00:52:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T00:52:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: python2.7-minimal 2024-04-26T00:52:30.822 INFO:teuthology.orchestra.run.smithi164.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T00:52:30.823 INFO:teuthology.orchestra.run.smithi164.stdout:The following additional packages will be installed: 2024-04-26T00:52:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T00:52:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T00:52:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T00:52:30.824 INFO:teuthology.orchestra.run.smithi164.stdout:Suggested packages: 2024-04-26T00:52:30.824 INFO:teuthology.orchestra.run.smithi164.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T00:52:30.824 INFO:teuthology.orchestra.run.smithi164.stdout:Recommended packages: 2024-04-26T00:52:30.824 INFO:teuthology.orchestra.run.smithi164.stdout: thermald 2024-04-26T00:52:30.884 INFO:teuthology.orchestra.run.smithi164.stdout:The following NEW packages will be installed: 2024-04-26T00:52:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T00:52:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T00:52:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T00:52:30.885 INFO:teuthology.orchestra.run.smithi164.stdout:The following packages will be upgraded: 2024-04-26T00:52:30.886 INFO:teuthology.orchestra.run.smithi164.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T00:52:31.014 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-04-26T00:52:31.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T00:52:31.066 INFO:teuthology.orchestra.run.smithi164.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T00:52:31.066 INFO:teuthology.orchestra.run.smithi164.stdout:Need to get 113 MB of archives. 2024-04-26T00:52:31.066 INFO:teuthology.orchestra.run.smithi164.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T00:52:31.066 INFO:teuthology.orchestra.run.smithi164.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T00:52:31.092 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-04-26T00:52:31.292 INFO:teuthology.orchestra.run.smithi116.stdout:Building dependency tree... 2024-04-26T00:52:31.293 INFO:teuthology.orchestra.run.smithi116.stdout:Reading state information... 2024-04-26T00:52:31.470 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T00:52:31.513 INFO:teuthology.orchestra.run.smithi116.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T00:52:31.513 INFO:teuthology.orchestra.run.smithi116.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T00:52:31.513 INFO:teuthology.orchestra.run.smithi116.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T00:52:31.513 INFO:teuthology.orchestra.run.smithi116.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T00:52:31.514 INFO:teuthology.orchestra.run.smithi116.stdout: python2.7-minimal 2024-04-26T00:52:31.514 INFO:teuthology.orchestra.run.smithi116.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T00:52:31.514 INFO:teuthology.orchestra.run.smithi116.stdout:The following additional packages will be installed: 2024-04-26T00:52:31.514 INFO:teuthology.orchestra.run.smithi116.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T00:52:31.514 INFO:teuthology.orchestra.run.smithi116.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T00:52:31.514 INFO:teuthology.orchestra.run.smithi116.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T00:52:31.514 INFO:teuthology.orchestra.run.smithi116.stdout:Suggested packages: 2024-04-26T00:52:31.514 INFO:teuthology.orchestra.run.smithi116.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T00:52:31.514 INFO:teuthology.orchestra.run.smithi116.stdout:Recommended packages: 2024-04-26T00:52:31.514 INFO:teuthology.orchestra.run.smithi116.stdout: thermald 2024-04-26T00:52:31.527 INFO:teuthology.orchestra.run.smithi116.stdout:The following NEW packages will be installed: 2024-04-26T00:52:31.528 INFO:teuthology.orchestra.run.smithi116.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T00:52:31.528 INFO:teuthology.orchestra.run.smithi116.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T00:52:31.528 INFO:teuthology.orchestra.run.smithi116.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T00:52:31.529 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages will be upgraded: 2024-04-26T00:52:31.530 INFO:teuthology.orchestra.run.smithi116.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T00:52:31.588 INFO:teuthology.orchestra.run.smithi116.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T00:52:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:Need to get 113 MB of archives. 2024-04-26T00:52:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T00:52:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T00:52:31.760 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T00:52:31.781 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:52:31.887 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:32.099 INFO:teuthology.orchestra.run.smithi116.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T00:52:32.227 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T00:52:32.355 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T00:52:32.376 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:52:32.413 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:32.668 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T00:52:32.689 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:52:32.937 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:33.213 INFO:teuthology.orchestra.run.smithi116.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T00:52:33.213 INFO:teuthology.orchestra.run.smithi116.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T00:52:33.213 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T00:52:33.342 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T00:52:33.372 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T00:52:33.394 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T00:52:33.404 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:52:33.445 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:33.787 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 113 MB in 2s (61.7 MB/s) 2024-04-26T00:52:33.955 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T00:52:35.930 INFO:teuthology.orchestra.run.smithi116.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T00:52:35.936 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:52:36.093 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:38.227 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T00:52:38.248 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:52:38.337 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:38.822 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T00:52:38.841 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:52:38.847 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:52:38.880 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:38.965 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:52:39.300 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:52:39.410 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:52:39.705 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T00:52:39.719 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T00:52:39.762 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T00:52:40.003 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:52:40.113 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:52:40.471 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:52:40.599 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:52:40.902 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T00:52:40.925 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T00:52:40.985 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T00:52:45.160 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T00:52:45.197 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:52:45.242 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:45.727 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:52:45.838 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:52:46.204 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:52:46.324 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:52:46.619 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T00:52:46.642 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T00:52:46.702 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T00:52:46.770 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T00:52:46.807 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:52:46.862 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:48.448 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:52:48.562 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:52:49.689 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T00:52:49.793 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:49.930 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T00:52:50.047 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:50.619 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:52:50.757 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:52:51.107 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T00:52:51.245 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:51.371 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T00:52:51.403 INFO:teuthology.orchestra.run.smithi012.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:52:51.404 INFO:teuthology.orchestra.run.smithi012.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T00:52:51.404 INFO:teuthology.orchestra.run.smithi012.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:52:51.404 INFO:teuthology.orchestra.run.smithi012.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T00:52:51.491 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:51.549 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:52.824 INFO:teuthology.orchestra.run.smithi008.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:52:52.824 INFO:teuthology.orchestra.run.smithi008.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T00:52:52.824 INFO:teuthology.orchestra.run.smithi008.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:52:52.824 INFO:teuthology.orchestra.run.smithi008.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T00:52:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T00:52:52.958 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:52:52.976 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:52.993 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:54.410 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T00:52:54.419 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:54.694 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T00:52:55.454 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:55.531 INFO:teuthology.orchestra.run.smithi012.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T00:52:55.531 INFO:teuthology.orchestra.run.smithi012.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:52:55.929 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T00:52:56.047 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T00:52:56.148 INFO:teuthology.orchestra.run.smithi008.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:56.219 INFO:teuthology.orchestra.run.smithi008.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T00:52:56.219 INFO:teuthology.orchestra.run.smithi008.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:52:56.250 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:52:56.381 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:52:56.714 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T00:52:56.836 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:56.962 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T00:52:57.107 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:58.474 INFO:teuthology.orchestra.run.smithi116.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:52:58.474 INFO:teuthology.orchestra.run.smithi116.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T00:52:58.475 INFO:teuthology.orchestra.run.smithi116.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:52:58.475 INFO:teuthology.orchestra.run.smithi116.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T00:52:58.625 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:52:59.994 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:53:01.455 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T00:53:01.581 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T00:53:01.699 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:53:01.783 INFO:teuthology.orchestra.run.smithi116.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T00:53:01.783 INFO:teuthology.orchestra.run.smithi116.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:53:09.581 INFO:teuthology.orchestra.run.smithi012.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T00:53:09.581 INFO:teuthology.orchestra.run.smithi012.stdout:Sourcing file `/etc/default/grub' 2024-04-26T00:53:09.588 INFO:teuthology.orchestra.run.smithi012.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T00:53:09.623 INFO:teuthology.orchestra.run.smithi012.stdout:Generating grub configuration file ... 2024-04-26T00:53:09.910 INFO:teuthology.orchestra.run.smithi012.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:53:09.945 INFO:teuthology.orchestra.run.smithi012.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:53:10.038 INFO:teuthology.orchestra.run.smithi164.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T00:53:10.233 INFO:teuthology.orchestra.run.smithi012.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:53:10.239 INFO:teuthology.orchestra.run.smithi012.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T00:53:10.309 INFO:teuthology.orchestra.run.smithi012.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T00:53:10.314 INFO:teuthology.orchestra.run.smithi012.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T00:53:10.466 INFO:teuthology.orchestra.run.smithi008.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T00:53:10.466 INFO:teuthology.orchestra.run.smithi008.stdout:Sourcing file `/etc/default/grub' 2024-04-26T00:53:10.473 INFO:teuthology.orchestra.run.smithi008.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T00:53:10.474 INFO:teuthology.orchestra.run.smithi012.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T00:53:10.474 INFO:teuthology.orchestra.run.smithi012.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T00:53:10.475 INFO:teuthology.orchestra.run.smithi012.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T00:53:10.493 INFO:teuthology.orchestra.run.smithi008.stdout:Generating grub configuration file ... 2024-04-26T00:53:10.516 INFO:teuthology.orchestra.run.smithi012.stdout:done 2024-04-26T00:53:10.769 INFO:teuthology.orchestra.run.smithi008.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:53:10.796 INFO:teuthology.orchestra.run.smithi008.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:53:11.021 INFO:teuthology.orchestra.run.smithi164.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T00:53:11.044 DEBUG:teuthology.orchestra.run.smithi012:> dpkg -s linux-image-generic 2024-04-26T00:53:11.073 INFO:teuthology.orchestra.run.smithi012.stdout:Package: linux-image-generic 2024-04-26T00:53:11.073 INFO:teuthology.orchestra.run.smithi012.stdout:Status: install ok installed 2024-04-26T00:53:11.073 INFO:teuthology.orchestra.run.smithi012.stdout:Priority: optional 2024-04-26T00:53:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:Section: kernel 2024-04-26T00:53:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:Installed-Size: 21 2024-04-26T00:53:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T00:53:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: amd64 2024-04-26T00:53:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:Source: linux-meta 2024-04-26T00:53:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:Version: 5.15.0.105.102 2024-04-26T00:53:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T00:53:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T00:53:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:Recommends: thermald 2024-04-26T00:53:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:Description: Generic Linux kernel image 2024-04-26T00:53:11.074 INFO:teuthology.orchestra.run.smithi012.stdout: This package will always depend on the latest generic kernel image 2024-04-26T00:53:11.074 INFO:teuthology.orchestra.run.smithi012.stdout: available. 2024-04-26T00:53:11.075 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T00:53:11.075 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T00:53:11.075 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T00:53:11.075 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-26T00:53:11.075 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T00:53:11.075 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get clean 2024-04-26T00:53:11.145 INFO:teuthology.orchestra.run.smithi008.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:53:11.150 INFO:teuthology.orchestra.run.smithi008.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T00:53:11.220 INFO:teuthology.orchestra.run.smithi008.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T00:53:11.226 INFO:teuthology.orchestra.run.smithi008.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T00:53:11.281 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get update 2024-04-26T00:53:11.385 INFO:teuthology.orchestra.run.smithi008.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T00:53:11.385 INFO:teuthology.orchestra.run.smithi008.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T00:53:11.386 INFO:teuthology.orchestra.run.smithi008.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T00:53:11.425 INFO:teuthology.orchestra.run.smithi008.stdout:done 2024-04-26T00:53:11.428 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T00:53:11.428 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T00:53:11.440 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T00:53:11.462 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T00:53:12.047 DEBUG:teuthology.orchestra.run.smithi008:> dpkg -s linux-image-generic 2024-04-26T00:53:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:Package: linux-image-generic 2024-04-26T00:53:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:Status: install ok installed 2024-04-26T00:53:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:Priority: optional 2024-04-26T00:53:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:Section: kernel 2024-04-26T00:53:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:Installed-Size: 21 2024-04-26T00:53:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T00:53:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:Architecture: amd64 2024-04-26T00:53:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:Source: linux-meta 2024-04-26T00:53:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:Version: 5.15.0.105.102 2024-04-26T00:53:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T00:53:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T00:53:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:Recommends: thermald 2024-04-26T00:53:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:Description: Generic Linux kernel image 2024-04-26T00:53:12.095 INFO:teuthology.orchestra.run.smithi008.stdout: This package will always depend on the latest generic kernel image 2024-04-26T00:53:12.095 INFO:teuthology.orchestra.run.smithi008.stdout: available. 2024-04-26T00:53:12.095 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T00:53:12.095 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T00:53:12.095 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T00:53:12.096 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-26T00:53:12.096 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi008.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T00:53:12.096 DEBUG:teuthology.orchestra.run.smithi008:> sudo apt-get clean 2024-04-26T00:53:12.249 DEBUG:teuthology.orchestra.run.smithi008:> sudo apt-get update 2024-04-26T00:53:12.386 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T00:53:12.386 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T00:53:12.400 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T00:53:12.421 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T00:53:12.905 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-04-26T00:53:12.924 DEBUG:teuthology.orchestra.run.smithi012:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T00:53:12.978 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-04-26T00:53:13.160 INFO:teuthology.orchestra.run.smithi012.stdout:Building dependency tree... 2024-04-26T00:53:13.161 INFO:teuthology.orchestra.run.smithi012.stdout:Reading state information... 2024-04-26T00:53:13.274 INFO:teuthology.orchestra.run.smithi164.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T00:53:13.326 INFO:teuthology.orchestra.run.smithi012.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T00:53:13.326 INFO:teuthology.orchestra.run.smithi012.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T00:53:13.326 INFO:teuthology.orchestra.run.smithi012.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T00:53:13.327 INFO:teuthology.orchestra.run.smithi012.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T00:53:13.327 INFO:teuthology.orchestra.run.smithi012.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T00:53:13.327 INFO:teuthology.orchestra.run.smithi012.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T00:53:13.327 INFO:teuthology.orchestra.run.smithi012.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T00:53:13.327 INFO:teuthology.orchestra.run.smithi012.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T00:53:13.356 INFO:teuthology.orchestra.run.smithi164.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T00:53:13.366 INFO:teuthology.orchestra.run.smithi012.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T00:53:13.368 DEBUG:teuthology.orchestra.run.smithi012:> dpkg -s linux-image-generic 2024-04-26T00:53:13.425 INFO:teuthology.orchestra.run.smithi012.stdout:Package: linux-image-generic 2024-04-26T00:53:13.425 INFO:teuthology.orchestra.run.smithi012.stdout:Status: install ok installed 2024-04-26T00:53:13.425 INFO:teuthology.orchestra.run.smithi012.stdout:Priority: optional 2024-04-26T00:53:13.425 INFO:teuthology.orchestra.run.smithi012.stdout:Section: kernel 2024-04-26T00:53:13.425 INFO:teuthology.orchestra.run.smithi012.stdout:Installed-Size: 21 2024-04-26T00:53:13.425 INFO:teuthology.orchestra.run.smithi012.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T00:53:13.425 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: amd64 2024-04-26T00:53:13.426 INFO:teuthology.orchestra.run.smithi012.stdout:Source: linux-meta 2024-04-26T00:53:13.426 INFO:teuthology.orchestra.run.smithi012.stdout:Version: 5.15.0.105.102 2024-04-26T00:53:13.426 INFO:teuthology.orchestra.run.smithi012.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T00:53:13.426 INFO:teuthology.orchestra.run.smithi012.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T00:53:13.426 INFO:teuthology.orchestra.run.smithi012.stdout:Recommends: thermald 2024-04-26T00:53:13.426 INFO:teuthology.orchestra.run.smithi012.stdout:Description: Generic Linux kernel image 2024-04-26T00:53:13.426 INFO:teuthology.orchestra.run.smithi012.stdout: This package will always depend on the latest generic kernel image 2024-04-26T00:53:13.426 INFO:teuthology.orchestra.run.smithi012.stdout: available. 2024-04-26T00:53:13.427 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-04-26T00:53:13.438 INFO:teuthology.orchestra.run.smithi164.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T00:53:13.474 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.Tkg6lTbFXB 2024-04-26T00:53:13.474 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Tkg6lTbFXB 2024-04-26T00:53:13.529 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.Tkg6lTbFXB 2024-04-26T00:53:13.648 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.Tkg6lTbFXB is 10KB 2024-04-26T00:53:13.661 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.Tkg6lTbFXB 2024-04-26T00:53:13.667 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T00:53:13.721 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T00:53:13.722 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T00:53:13.722 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T00:53:13.789 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T00:53:13.789 DEBUG:teuthology.orchestra.run.smithi012:> sudo update-grub 2024-04-26T00:53:13.851 INFO:teuthology.orchestra.run.smithi008.stdout:Reading package lists... 2024-04-26T00:53:13.868 DEBUG:teuthology.orchestra.run.smithi008:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T00:53:13.898 INFO:teuthology.orchestra.run.smithi164.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T00:53:13.921 INFO:teuthology.orchestra.run.smithi008.stdout:Reading package lists... 2024-04-26T00:53:14.055 INFO:teuthology.orchestra.run.smithi164.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T00:53:14.122 INFO:teuthology.orchestra.run.smithi008.stdout:Building dependency tree... 2024-04-26T00:53:14.122 INFO:teuthology.orchestra.run.smithi008.stdout:Reading state information... 2024-04-26T00:53:14.284 INFO:teuthology.orchestra.run.smithi008.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T00:53:14.284 INFO:teuthology.orchestra.run.smithi008.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T00:53:14.284 INFO:teuthology.orchestra.run.smithi008.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T00:53:14.284 INFO:teuthology.orchestra.run.smithi008.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T00:53:14.285 INFO:teuthology.orchestra.run.smithi008.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T00:53:14.285 INFO:teuthology.orchestra.run.smithi008.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T00:53:14.285 INFO:teuthology.orchestra.run.smithi008.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T00:53:14.285 INFO:teuthology.orchestra.run.smithi008.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T00:53:14.321 INFO:teuthology.orchestra.run.smithi008.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T00:53:14.323 DEBUG:teuthology.orchestra.run.smithi008:> dpkg -s linux-image-generic 2024-04-26T00:53:14.380 INFO:teuthology.orchestra.run.smithi008.stdout:Package: linux-image-generic 2024-04-26T00:53:14.380 INFO:teuthology.orchestra.run.smithi008.stdout:Status: install ok installed 2024-04-26T00:53:14.380 INFO:teuthology.orchestra.run.smithi008.stdout:Priority: optional 2024-04-26T00:53:14.380 INFO:teuthology.orchestra.run.smithi008.stdout:Section: kernel 2024-04-26T00:53:14.380 INFO:teuthology.orchestra.run.smithi008.stdout:Installed-Size: 21 2024-04-26T00:53:14.380 INFO:teuthology.orchestra.run.smithi008.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T00:53:14.380 INFO:teuthology.orchestra.run.smithi008.stdout:Architecture: amd64 2024-04-26T00:53:14.380 INFO:teuthology.orchestra.run.smithi008.stdout:Source: linux-meta 2024-04-26T00:53:14.381 INFO:teuthology.orchestra.run.smithi008.stdout:Version: 5.15.0.105.102 2024-04-26T00:53:14.381 INFO:teuthology.orchestra.run.smithi008.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T00:53:14.381 INFO:teuthology.orchestra.run.smithi008.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T00:53:14.381 INFO:teuthology.orchestra.run.smithi008.stdout:Recommends: thermald 2024-04-26T00:53:14.381 INFO:teuthology.orchestra.run.smithi008.stdout:Description: Generic Linux kernel image 2024-04-26T00:53:14.381 INFO:teuthology.orchestra.run.smithi008.stdout: This package will always depend on the latest generic kernel image 2024-04-26T00:53:14.381 INFO:teuthology.orchestra.run.smithi008.stdout: available. 2024-04-26T00:53:14.381 DEBUG:teuthology.orchestra.run.smithi008:> mktemp 2024-04-26T00:53:14.428 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.eUCfgsT9Kv 2024-04-26T00:53:14.429 DEBUG:teuthology.orchestra.run.smithi008:> sudo cp /boot/grub/grub.cfg /tmp/tmp.eUCfgsT9Kv 2024-04-26T00:53:14.466 INFO:teuthology.orchestra.run.smithi164.stdout:Fetched 113 MB in 43s (2,625 kB/s) 2024-04-26T00:53:14.484 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.eUCfgsT9Kv 2024-04-26T00:53:14.616 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.eUCfgsT9Kv is 10KB 2024-04-26T00:53:14.629 DEBUG:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.eUCfgsT9Kv 2024-04-26T00:53:14.632 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T00:53:14.635 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T00:53:14.689 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T00:53:14.689 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T00:53:14.689 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T00:53:14.773 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T00:53:14.773 DEBUG:teuthology.orchestra.run.smithi008:> sudo update-grub 2024-04-26T00:53:14.971 INFO:teuthology.orchestra.run.smithi012.stderr:Sourcing file `/etc/default/grub' 2024-04-26T00:53:14.972 INFO:teuthology.orchestra.run.smithi012.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T00:53:14.975 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2024-04-26T00:53:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:53:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:53:15.370 INFO:teuthology.orchestra.run.smithi012.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:53:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T00:53:15.418 INFO:teuthology.orchestra.run.smithi012.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T00:53:15.423 INFO:teuthology.orchestra.run.smithi012.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T00:53:15.584 INFO:teuthology.orchestra.run.smithi012.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T00:53:15.585 INFO:teuthology.orchestra.run.smithi012.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T00:53:15.585 INFO:teuthology.orchestra.run.smithi012.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T00:53:15.604 INFO:teuthology.orchestra.run.smithi012.stderr:done 2024-04-26T00:53:15.613 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2024-04-26T00:53:15.983 INFO:teuthology.orchestra.run.smithi116.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T00:53:15.984 INFO:teuthology.orchestra.run.smithi116.stdout:Sourcing file `/etc/default/grub' 2024-04-26T00:53:15.994 INFO:teuthology.orchestra.run.smithi116.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T00:53:16.003 INFO:teuthology.orchestra.run.smithi116.stdout:Generating grub configuration file ... 2024-04-26T00:53:16.211 INFO:teuthology.orchestra.run.smithi008.stderr:Sourcing file `/etc/default/grub' 2024-04-26T00:53:16.212 INFO:teuthology.orchestra.run.smithi008.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T00:53:16.215 INFO:teuthology.orchestra.run.smithi008.stderr:Generating grub configuration file ... 2024-04-26T00:53:16.313 INFO:teuthology.orchestra.run.smithi116.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:53:16.336 INFO:teuthology.orchestra.run.smithi116.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:53:16.458 INFO:teuthology.orchestra.run.smithi008.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:53:16.464 INFO:teuthology.orchestra.run.smithi008.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:53:16.611 INFO:teuthology.orchestra.run.smithi116.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:53:16.617 INFO:teuthology.orchestra.run.smithi116.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T00:53:16.635 INFO:teuthology.orchestra.run.smithi164.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T00:53:16.639 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:53:16.684 INFO:teuthology.orchestra.run.smithi116.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T00:53:16.690 INFO:teuthology.orchestra.run.smithi116.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T00:53:16.704 INFO:teuthology.orchestra.run.smithi008.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:53:16.710 INFO:teuthology.orchestra.run.smithi008.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T00:53:16.745 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:53:16.756 INFO:teuthology.orchestra.run.smithi008.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T00:53:16.761 INFO:teuthology.orchestra.run.smithi008.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T00:53:16.860 INFO:teuthology.orchestra.run.smithi116.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T00:53:16.860 INFO:teuthology.orchestra.run.smithi116.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T00:53:16.860 INFO:teuthology.orchestra.run.smithi116.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T00:53:16.887 INFO:teuthology.orchestra.run.smithi008.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T00:53:16.887 INFO:teuthology.orchestra.run.smithi008.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T00:53:16.887 INFO:teuthology.orchestra.run.smithi008.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T00:53:16.897 INFO:teuthology.orchestra.run.smithi008.stderr:done 2024-04-26T00:53:16.898 DEBUG:teuthology.orchestra.run.smithi008:> sudo shutdown -r now 2024-04-26T00:53:16.911 INFO:teuthology.orchestra.run.smithi116.stdout:done 2024-04-26T00:53:17.420 DEBUG:teuthology.orchestra.run.smithi116:> dpkg -s linux-image-generic 2024-04-26T00:53:17.449 INFO:teuthology.orchestra.run.smithi116.stdout:Package: linux-image-generic 2024-04-26T00:53:17.449 INFO:teuthology.orchestra.run.smithi116.stdout:Status: install ok installed 2024-04-26T00:53:17.449 INFO:teuthology.orchestra.run.smithi116.stdout:Priority: optional 2024-04-26T00:53:17.449 INFO:teuthology.orchestra.run.smithi116.stdout:Section: kernel 2024-04-26T00:53:17.449 INFO:teuthology.orchestra.run.smithi116.stdout:Installed-Size: 21 2024-04-26T00:53:17.449 INFO:teuthology.orchestra.run.smithi116.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T00:53:17.449 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: amd64 2024-04-26T00:53:17.449 INFO:teuthology.orchestra.run.smithi116.stdout:Source: linux-meta 2024-04-26T00:53:17.449 INFO:teuthology.orchestra.run.smithi116.stdout:Version: 5.15.0.105.102 2024-04-26T00:53:17.450 INFO:teuthology.orchestra.run.smithi116.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T00:53:17.450 INFO:teuthology.orchestra.run.smithi116.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T00:53:17.450 INFO:teuthology.orchestra.run.smithi116.stdout:Recommends: thermald 2024-04-26T00:53:17.450 INFO:teuthology.orchestra.run.smithi116.stdout:Description: Generic Linux kernel image 2024-04-26T00:53:17.450 INFO:teuthology.orchestra.run.smithi116.stdout: This package will always depend on the latest generic kernel image 2024-04-26T00:53:17.450 INFO:teuthology.orchestra.run.smithi116.stdout: available. 2024-04-26T00:53:17.450 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T00:53:17.450 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T00:53:17.450 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T00:53:17.451 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-26T00:53:17.451 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T00:53:17.451 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get clean 2024-04-26T00:53:17.652 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get update 2024-04-26T00:53:17.805 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T00:53:17.805 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T00:53:17.814 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T00:53:17.835 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T00:53:19.021 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T00:53:19.042 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:53:19.136 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:53:19.186 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-04-26T00:53:19.206 DEBUG:teuthology.orchestra.run.smithi116:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T00:53:19.258 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-04-26T00:53:19.458 INFO:teuthology.orchestra.run.smithi116.stdout:Building dependency tree... 2024-04-26T00:53:19.460 INFO:teuthology.orchestra.run.smithi116.stdout:Reading state information... 2024-04-26T00:53:19.621 INFO:teuthology.orchestra.run.smithi116.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T00:53:19.622 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T00:53:19.622 INFO:teuthology.orchestra.run.smithi116.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T00:53:19.622 INFO:teuthology.orchestra.run.smithi116.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T00:53:19.622 INFO:teuthology.orchestra.run.smithi116.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T00:53:19.622 INFO:teuthology.orchestra.run.smithi116.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T00:53:19.623 INFO:teuthology.orchestra.run.smithi116.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T00:53:19.623 INFO:teuthology.orchestra.run.smithi116.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T00:53:19.632 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T00:53:19.653 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:53:19.661 INFO:teuthology.orchestra.run.smithi116.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T00:53:19.663 DEBUG:teuthology.orchestra.run.smithi116:> dpkg -s linux-image-generic 2024-04-26T00:53:19.690 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:53:19.716 INFO:teuthology.orchestra.run.smithi116.stdout:Package: linux-image-generic 2024-04-26T00:53:19.717 INFO:teuthology.orchestra.run.smithi116.stdout:Status: install ok installed 2024-04-26T00:53:19.717 INFO:teuthology.orchestra.run.smithi116.stdout:Priority: optional 2024-04-26T00:53:19.717 INFO:teuthology.orchestra.run.smithi116.stdout:Section: kernel 2024-04-26T00:53:19.717 INFO:teuthology.orchestra.run.smithi116.stdout:Installed-Size: 21 2024-04-26T00:53:19.717 INFO:teuthology.orchestra.run.smithi116.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T00:53:19.717 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: amd64 2024-04-26T00:53:19.717 INFO:teuthology.orchestra.run.smithi116.stdout:Source: linux-meta 2024-04-26T00:53:19.717 INFO:teuthology.orchestra.run.smithi116.stdout:Version: 5.15.0.105.102 2024-04-26T00:53:19.717 INFO:teuthology.orchestra.run.smithi116.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T00:53:19.717 INFO:teuthology.orchestra.run.smithi116.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T00:53:19.717 INFO:teuthology.orchestra.run.smithi116.stdout:Recommends: thermald 2024-04-26T00:53:19.717 INFO:teuthology.orchestra.run.smithi116.stdout:Description: Generic Linux kernel image 2024-04-26T00:53:19.718 INFO:teuthology.orchestra.run.smithi116.stdout: This package will always depend on the latest generic kernel image 2024-04-26T00:53:19.718 INFO:teuthology.orchestra.run.smithi116.stdout: available. 2024-04-26T00:53:19.718 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-04-26T00:53:19.762 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.T5RPBJmuqK 2024-04-26T00:53:19.762 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub/grub.cfg /tmp/tmp.T5RPBJmuqK 2024-04-26T00:53:19.819 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.T5RPBJmuqK 2024-04-26T00:53:19.950 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.T5RPBJmuqK is 10KB 2024-04-26T00:53:19.963 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.T5RPBJmuqK 2024-04-26T00:53:19.968 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T00:53:20.021 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T00:53:20.021 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T00:53:20.021 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T00:53:20.097 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T00:53:20.097 DEBUG:teuthology.orchestra.run.smithi116:> sudo update-grub 2024-04-26T00:53:21.290 INFO:teuthology.orchestra.run.smithi116.stderr:Sourcing file `/etc/default/grub' 2024-04-26T00:53:21.291 INFO:teuthology.orchestra.run.smithi116.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T00:53:21.293 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-04-26T00:53:21.497 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:53:21.504 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:53:21.798 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:53:21.804 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T00:53:21.848 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T00:53:21.853 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T00:53:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T00:53:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T00:53:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T00:53:21.985 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-04-26T00:53:21.988 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-04-26T00:53:26.615 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:53:26.776 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:53:27.142 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:53:27.457 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:53:27.599 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T00:53:27.621 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T00:53:27.657 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T00:53:33.462 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T00:53:33.499 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:53:33.536 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:53:36.889 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:53:37.005 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:53:37.372 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T00:53:37.485 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:53:37.611 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T00:53:37.739 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:53:39.083 INFO:teuthology.orchestra.run.smithi164.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:53:39.083 INFO:teuthology.orchestra.run.smithi164.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T00:53:39.084 INFO:teuthology.orchestra.run.smithi164.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:53:39.084 INFO:teuthology.orchestra.run.smithi164.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T00:53:39.244 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:53:40.663 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:53:42.063 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T00:53:42.214 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T00:53:42.340 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:53:42.402 INFO:teuthology.orchestra.run.smithi164.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T00:53:42.402 INFO:teuthology.orchestra.run.smithi164.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:53:45.616 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-26T00:53:45.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-26T00:53:45.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:53:46.942 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-26T00:53:46.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-04-26T00:53:46.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:53:51.990 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-26T00:53:51.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-04-26T00:53:51.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:53:56.784 INFO:teuthology.orchestra.run.smithi164.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T00:53:56.785 INFO:teuthology.orchestra.run.smithi164.stdout:Sourcing file `/etc/default/grub' 2024-04-26T00:53:56.817 INFO:teuthology.orchestra.run.smithi164.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T00:53:56.880 INFO:teuthology.orchestra.run.smithi164.stdout:Generating grub configuration file ... 2024-04-26T00:53:57.239 INFO:teuthology.orchestra.run.smithi164.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:53:57.260 INFO:teuthology.orchestra.run.smithi164.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:53:57.681 INFO:teuthology.orchestra.run.smithi164.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:53:57.686 INFO:teuthology.orchestra.run.smithi164.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T00:53:57.758 INFO:teuthology.orchestra.run.smithi164.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T00:53:57.763 INFO:teuthology.orchestra.run.smithi164.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T00:53:57.964 INFO:teuthology.orchestra.run.smithi164.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T00:53:57.964 INFO:teuthology.orchestra.run.smithi164.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T00:53:57.965 INFO:teuthology.orchestra.run.smithi164.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T00:53:58.017 INFO:teuthology.orchestra.run.smithi164.stdout:done 2024-04-26T00:53:58.492 DEBUG:teuthology.orchestra.run.smithi164:> dpkg -s linux-image-generic 2024-04-26T00:53:58.515 INFO:teuthology.orchestra.run.smithi164.stdout:Package: linux-image-generic 2024-04-26T00:53:58.515 INFO:teuthology.orchestra.run.smithi164.stdout:Status: install ok installed 2024-04-26T00:53:58.515 INFO:teuthology.orchestra.run.smithi164.stdout:Priority: optional 2024-04-26T00:53:58.515 INFO:teuthology.orchestra.run.smithi164.stdout:Section: kernel 2024-04-26T00:53:58.515 INFO:teuthology.orchestra.run.smithi164.stdout:Installed-Size: 21 2024-04-26T00:53:58.516 INFO:teuthology.orchestra.run.smithi164.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T00:53:58.516 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: amd64 2024-04-26T00:53:58.516 INFO:teuthology.orchestra.run.smithi164.stdout:Source: linux-meta 2024-04-26T00:53:58.516 INFO:teuthology.orchestra.run.smithi164.stdout:Version: 5.15.0.105.102 2024-04-26T00:53:58.516 INFO:teuthology.orchestra.run.smithi164.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T00:53:58.516 INFO:teuthology.orchestra.run.smithi164.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T00:53:58.516 INFO:teuthology.orchestra.run.smithi164.stdout:Recommends: thermald 2024-04-26T00:53:58.516 INFO:teuthology.orchestra.run.smithi164.stdout:Description: Generic Linux kernel image 2024-04-26T00:53:58.516 INFO:teuthology.orchestra.run.smithi164.stdout: This package will always depend on the latest generic kernel image 2024-04-26T00:53:58.516 INFO:teuthology.orchestra.run.smithi164.stdout: available. 2024-04-26T00:53:58.517 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T00:53:58.517 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T00:53:58.517 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T00:53:58.517 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-04-26T00:53:58.517 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi164.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T00:53:58.517 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-get clean 2024-04-26T00:53:58.654 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-get update 2024-04-26T00:53:58.780 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T00:53:58.892 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T00:53:58.981 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T00:53:59.073 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T00:54:00.204 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2024-04-26T00:54:00.221 DEBUG:teuthology.orchestra.run.smithi164:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T00:54:00.290 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2024-04-26T00:54:00.493 INFO:teuthology.orchestra.run.smithi164.stdout:Building dependency tree... 2024-04-26T00:54:00.493 INFO:teuthology.orchestra.run.smithi164.stdout:Reading state information... 2024-04-26T00:54:00.669 INFO:teuthology.orchestra.run.smithi164.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T00:54:00.669 INFO:teuthology.orchestra.run.smithi164.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T00:54:00.669 INFO:teuthology.orchestra.run.smithi164.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T00:54:00.669 INFO:teuthology.orchestra.run.smithi164.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T00:54:00.669 INFO:teuthology.orchestra.run.smithi164.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T00:54:00.669 INFO:teuthology.orchestra.run.smithi164.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T00:54:00.670 INFO:teuthology.orchestra.run.smithi164.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T00:54:00.670 INFO:teuthology.orchestra.run.smithi164.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T00:54:00.713 INFO:teuthology.orchestra.run.smithi164.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T00:54:00.714 DEBUG:teuthology.orchestra.run.smithi164:> dpkg -s linux-image-generic 2024-04-26T00:54:00.729 INFO:teuthology.orchestra.run.smithi164.stdout:Package: linux-image-generic 2024-04-26T00:54:00.729 INFO:teuthology.orchestra.run.smithi164.stdout:Status: install ok installed 2024-04-26T00:54:00.729 INFO:teuthology.orchestra.run.smithi164.stdout:Priority: optional 2024-04-26T00:54:00.730 INFO:teuthology.orchestra.run.smithi164.stdout:Section: kernel 2024-04-26T00:54:00.730 INFO:teuthology.orchestra.run.smithi164.stdout:Installed-Size: 21 2024-04-26T00:54:00.730 INFO:teuthology.orchestra.run.smithi164.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T00:54:00.730 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: amd64 2024-04-26T00:54:00.730 INFO:teuthology.orchestra.run.smithi164.stdout:Source: linux-meta 2024-04-26T00:54:00.730 INFO:teuthology.orchestra.run.smithi164.stdout:Version: 5.15.0.105.102 2024-04-26T00:54:00.730 INFO:teuthology.orchestra.run.smithi164.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T00:54:00.730 INFO:teuthology.orchestra.run.smithi164.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T00:54:00.730 INFO:teuthology.orchestra.run.smithi164.stdout:Recommends: thermald 2024-04-26T00:54:00.730 INFO:teuthology.orchestra.run.smithi164.stdout:Description: Generic Linux kernel image 2024-04-26T00:54:00.730 INFO:teuthology.orchestra.run.smithi164.stdout: This package will always depend on the latest generic kernel image 2024-04-26T00:54:00.730 INFO:teuthology.orchestra.run.smithi164.stdout: available. 2024-04-26T00:54:00.731 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-04-26T00:54:00.776 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.GnUXorqwxW 2024-04-26T00:54:00.776 DEBUG:teuthology.orchestra.run.smithi164:> sudo cp /boot/grub/grub.cfg /tmp/tmp.GnUXorqwxW 2024-04-26T00:54:00.833 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.GnUXorqwxW 2024-04-26T00:54:00.957 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.GnUXorqwxW is 10KB 2024-04-26T00:54:00.974 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.GnUXorqwxW 2024-04-26T00:54:00.982 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T00:54:01.037 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T00:54:01.038 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T00:54:01.038 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T00:54:01.137 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T00:54:01.137 DEBUG:teuthology.orchestra.run.smithi164:> sudo update-grub 2024-04-26T00:54:02.343 INFO:teuthology.orchestra.run.smithi164.stderr:Sourcing file `/etc/default/grub' 2024-04-26T00:54:02.344 INFO:teuthology.orchestra.run.smithi164.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T00:54:02.347 INFO:teuthology.orchestra.run.smithi164.stderr:Generating grub configuration file ... 2024-04-26T00:54:02.615 INFO:teuthology.orchestra.run.smithi164.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:54:02.621 INFO:teuthology.orchestra.run.smithi164.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:54:02.952 INFO:teuthology.orchestra.run.smithi164.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:54:02.958 INFO:teuthology.orchestra.run.smithi164.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T00:54:03.004 INFO:teuthology.orchestra.run.smithi164.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T00:54:03.010 INFO:teuthology.orchestra.run.smithi164.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T00:54:03.173 INFO:teuthology.orchestra.run.smithi164.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T00:54:03.173 INFO:teuthology.orchestra.run.smithi164.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T00:54:03.173 INFO:teuthology.orchestra.run.smithi164.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T00:54:03.186 INFO:teuthology.orchestra.run.smithi164.stderr:done 2024-04-26T00:54:03.200 DEBUG:teuthology.orchestra.run.smithi164:> sudo shutdown -r now 2024-04-26T00:54:26.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-04-26T00:54:33.205 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-04-26T00:54:33.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-04-26T00:54:33.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:54:35.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-04-26T00:54:35.593 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:54:45.630 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T00:54:46.946 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T00:54:53.986 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-04-26T00:54:54.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-26T00:54:54.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:54:54.849 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-04-26T00:54:55.398 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-26T00:54:55.398 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-26T00:54:55.398 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-04-26T00:54:55.447 INFO:teuthology.orchestra.run.smithi012.stdout:5.15.0-105-generic 2024-04-26T00:54:55.447 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T00:54:55.447 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T00:54:55.447 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T00:54:55.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-04-26T00:54:55.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:54:56.275 DEBUG:teuthology.orchestra.run.smithi008:> true 2024-04-26T00:54:56.448 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-26T00:54:56.448 DEBUG:teuthology.orchestra.run.smithi012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T00:54:56.519 INFO:teuthology.orchestra.run.smithi012.stdout:ttyS1 2024-04-26T00:54:56.544 DEBUG:teuthology.parallel:result is None 2024-04-26T00:54:57.063 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-04-26T00:54:57.063 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-26T00:54:57.064 DEBUG:teuthology.orchestra.run.smithi008:> uname -r 2024-04-26T00:54:57.115 INFO:teuthology.orchestra.run.smithi008.stdout:5.15.0-105-generic 2024-04-26T00:54:57.115 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T00:54:57.115 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T00:54:57.115 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T00:54:58.116 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-26T00:54:58.116 DEBUG:teuthology.orchestra.run.smithi008:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T00:54:58.309 INFO:teuthology.orchestra.run.smithi008.stdout:ttyS1 2024-04-26T00:54:58.342 DEBUG:teuthology.parallel:result is None 2024-04-26T00:55:05.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-04-26T00:55:05.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:55:05.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-04-26T00:55:07.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-04-26T00:55:16.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-04-26T00:55:16.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:55:20.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-04-26T00:55:20.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:55:21.314 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-04-26T00:55:21.855 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-04-26T00:55:21.855 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.15.0-105-generic"... 2024-04-26T00:55:21.856 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-04-26T00:55:21.901 INFO:teuthology.orchestra.run.smithi116.stdout:5.15.0-105-generic 2024-04-26T00:55:21.902 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T00:55:21.902 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T00:55:21.902 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T00:55:22.903 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-26T00:55:22.903 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T00:55:22.982 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2024-04-26T00:55:23.031 DEBUG:teuthology.parallel:result is None 2024-04-26T00:56:16.811 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T00:56:28.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-04-26T00:56:28.813 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:56:29.121 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-04-26T00:56:29.682 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-04-26T00:56:29.682 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.15.0-105-generic"... 2024-04-26T00:56:29.682 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-04-26T00:56:29.727 INFO:teuthology.orchestra.run.smithi164.stdout:5.15.0-105-generic 2024-04-26T00:56:29.727 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T00:56:29.727 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T00:56:29.727 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T00:56:30.728 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-04-26T00:56:30.729 DEBUG:teuthology.orchestra.run.smithi164:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T00:56:30.820 INFO:teuthology.orchestra.run.smithi164.stdout:ttyS1 2024-04-26T00:56:30.837 DEBUG:teuthology.parallel:result is None 2024-04-26T00:56:30.838 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T00:56:30.848 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T00:56:30.848 DEBUG:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T00:56:30.851 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T00:56:30.853 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T00:56:30.856 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T00:56:30.862 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T00:56:30.889 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T00:56:30.921 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T00:56:30.922 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T00:56:30.924 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T00:56:30.926 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T00:56:30.929 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T00:56:31.034 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T00:56:31.041 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T00:56:31.041 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T00:56:31.044 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T00:56:31.046 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T00:56:31.048 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T00:56:31.061 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T00:56:31.063 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T00:56:31.065 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T00:56:31.067 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T00:56:31.069 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T00:56:31.071 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T00:56:31.074 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T00:56:31.075 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T00:56:31.077 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T00:56:31.084 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T00:56:31.084 DEBUG:teuthology.orchestra.run.smithi008:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T00:56:31.113 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T00:56:31.124 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T00:56:31.126 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T00:56:31.139 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T00:56:31.148 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T00:56:31.148 DEBUG:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T00:56:31.169 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T00:56:31.180 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T00:56:31.182 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T00:56:31.187 DEBUG:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T00:56:31.220 DEBUG:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T00:56:31.268 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T00:56:31.268 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T00:56:31.333 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T00:56:31.339 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T00:56:31.387 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T00:56:31.387 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T00:56:31.445 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T00:56:31.451 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T00:56:31.498 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T00:56:31.498 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T00:56:31.572 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T00:56:31.578 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T00:56:31.623 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T00:56:31.623 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T00:56:31.693 DEBUG:teuthology.orchestra.run.smithi008:> sudo service rsyslog restart 2024-04-26T00:56:31.695 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2024-04-26T00:56:31.698 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-04-26T00:56:31.700 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2024-04-26T00:56:31.763 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T00:56:31.770 INFO:teuthology.task.internal:Starting timer... 2024-04-26T00:56:31.770 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T00:56:31.801 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T00:56:31.832 DEBUG:teuthology.task.selinux:Excluding smithi008: OS 'ubuntu' does not support SELinux 2024-04-26T00:56:31.832 DEBUG:teuthology.task.selinux:Excluding smithi012: OS 'ubuntu' does not support SELinux 2024-04-26T00:56:31.833 DEBUG:teuthology.task.selinux:Excluding smithi116: OS 'ubuntu' does not support SELinux 2024-04-26T00:56:31.833 DEBUG:teuthology.task.selinux:Excluding smithi164: OS 'ubuntu' does not support SELinux 2024-04-26T00:56:31.833 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T00:56:31.833 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-26T00:56:31.833 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T00:56:31.833 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T00:56:31.866 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T00:56:31.884 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T00:56:31.885 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi008.front.sepia.ceph.com,smithi012.front.sepia.ceph.com,smithi116.front.sepia.ceph.com,smithi164.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T01:03:59.343 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi008.front.sepia.ceph.com'), Remote(name='ubuntu@smithi012.front.sepia.ceph.com'), Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi164.front.sepia.ceph.com')] 2024-04-26T01:03:59.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-04-26T01:03:59.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:03:59.438 DEBUG:teuthology.orchestra.run.smithi008:> true 2024-04-26T01:03:59.518 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-04-26T01:03:59.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-26T01:03:59.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:03:59.600 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-04-26T01:03:59.679 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-26T01:03:59.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-04-26T01:03:59.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:03:59.766 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-04-26T01:03:59.842 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-04-26T01:03:59.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-04-26T01:03:59.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:03:59.924 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-04-26T01:03:59.998 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-04-26T01:03:59.999 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T01:04:00.010 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T01:04:00.010 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:04:00.011 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:04:00.013 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:04:00.013 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:04:00.016 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:04:00.016 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:04:00.018 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:04:00.018 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:04:00.040 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T01:04:00.040 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: Command line: ntpd -gq 2024-04-26T01:04:00.040 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: ---------------------------------------------------- 2024-04-26T01:04:00.040 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T01:04:00.040 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T01:04:00.040 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: corporation. Support and training for ntp-4 are 2024-04-26T01:04:00.040 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: available at https://www.nwtime.org/support 2024-04-26T01:04:00.040 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: ---------------------------------------------------- 2024-04-26T01:04:00.041 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: proto: precision = 0.048 usec (-24) 2024-04-26T01:04:00.041 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: basedate set to 2022-02-04 2024-04-26T01:04:00.041 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: gps base set to 2022-02-06 (week 2196) 2024-04-26T01:04:00.041 INFO:teuthology.orchestra.run.smithi008.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:04:00.042 INFO:teuthology.orchestra.run.smithi008.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:04:00.043 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:04:00.043 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:04:00.043 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T01:04:00.043 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T01:04:00.043 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T01:04:00.043 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: Listen normally on 3 ens1f0 172.21.15.8:123 2024-04-26T01:04:00.043 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: Listen normally on 4 lo [::1]:123 2024-04-26T01:04:00.043 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:25fe%4]:123 2024-04-26T01:04:00.043 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:00 ntpd[17095]: Listening on routing socket on fd #22 for interface updates 2024-04-26T01:04:00.044 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T01:04:00.045 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: Command line: ntpd -gq 2024-04-26T01:04:00.045 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: ---------------------------------------------------- 2024-04-26T01:04:00.045 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T01:04:00.045 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T01:04:00.045 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: corporation. Support and training for ntp-4 are 2024-04-26T01:04:00.045 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: available at https://www.nwtime.org/support 2024-04-26T01:04:00.045 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: ---------------------------------------------------- 2024-04-26T01:04:00.045 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: proto: precision = 0.046 usec (-24) 2024-04-26T01:04:00.045 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: basedate set to 2022-02-04 2024-04-26T01:04:00.045 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: gps base set to 2022-02-06 (week 2196) 2024-04-26T01:04:00.046 INFO:teuthology.orchestra.run.smithi012.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:04:00.046 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:04:00.046 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:04:00.046 INFO:teuthology.orchestra.run.smithi012.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:04:00.047 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T01:04:00.047 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T01:04:00.047 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T01:04:00.047 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: Listen normally on 3 ens1f0 172.21.15.12:123 2024-04-26T01:04:00.047 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: Listen normally on 4 lo [::1]:123 2024-04-26T01:04:00.048 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:25ea%3]:123 2024-04-26T01:04:00.048 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:00 ntpd[17144]: Listening on routing socket on fd #22 for interface updates 2024-04-26T01:04:00.049 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T01:04:00.049 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: Command line: ntpd -gq 2024-04-26T01:04:00.049 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: ---------------------------------------------------- 2024-04-26T01:04:00.049 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T01:04:00.049 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T01:04:00.049 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: corporation. Support and training for ntp-4 are 2024-04-26T01:04:00.049 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: available at https://www.nwtime.org/support 2024-04-26T01:04:00.049 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: ---------------------------------------------------- 2024-04-26T01:04:00.050 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: proto: precision = 0.048 usec (-24) 2024-04-26T01:04:00.050 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: basedate set to 2022-02-04 2024-04-26T01:04:00.050 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: gps base set to 2022-02-06 (week 2196) 2024-04-26T01:04:00.050 INFO:teuthology.orchestra.run.smithi116.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:04:00.051 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:04:00.051 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:04:00.051 INFO:teuthology.orchestra.run.smithi116.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:04:00.052 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T01:04:00.052 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T01:04:00.052 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T01:04:00.052 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: Listen normally on 3 enp3s0f1 172.21.15.116:123 2024-04-26T01:04:00.052 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: Listen normally on 4 lo [::1]:123 2024-04-26T01:04:00.052 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:3f67%5]:123 2024-04-26T01:04:00.052 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:00 ntpd[17205]: Listening on routing socket on fd #22 for interface updates 2024-04-26T01:04:00.072 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T01:04:00.072 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: Command line: ntpd -gq 2024-04-26T01:04:00.072 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: ---------------------------------------------------- 2024-04-26T01:04:00.072 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T01:04:00.073 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T01:04:00.073 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: corporation. Support and training for ntp-4 are 2024-04-26T01:04:00.073 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: available at https://www.nwtime.org/support 2024-04-26T01:04:00.073 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: ---------------------------------------------------- 2024-04-26T01:04:00.073 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: proto: precision = 0.060 usec (-24) 2024-04-26T01:04:00.074 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: basedate set to 2022-02-04 2024-04-26T01:04:00.074 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: gps base set to 2022-02-06 (week 2196) 2024-04-26T01:04:00.074 INFO:teuthology.orchestra.run.smithi164.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:04:00.074 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:04:00.074 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:04:00.075 INFO:teuthology.orchestra.run.smithi164.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:04:00.075 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T01:04:00.076 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T01:04:00.076 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T01:04:00.076 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: Listen normally on 3 enp3s0f1 172.21.15.164:123 2024-04-26T01:04:00.076 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: Listen normally on 4 lo [::1]:123 2024-04-26T01:04:00.076 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:73f3%5]:123 2024-04-26T01:04:00.076 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:00 ntpd[17157]: Listening on routing socket on fd #22 for interface updates 2024-04-26T01:04:01.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:01 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:01.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:01 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:01.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:01 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:01.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:01 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:01.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:01 ntpd[17144]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:01.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:01 ntpd[17144]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:01.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:01 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:01.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:01 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:01.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:01 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:01.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:01 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:01.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:01 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:01.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:01 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:01.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:01 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:01.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:01 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:01.073 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:01 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:01.073 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:01 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:01.073 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:01 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:01.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:01 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:01.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:01 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:01.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:01 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:01.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:01 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:01.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:01 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:01.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:01 ntpd[17157]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:01.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:01 ntpd[17157]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:01.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:01 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:01.075 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:01 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:01.075 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:01 ntpd[17157]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:01.075 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:01 ntpd[17157]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:02.040 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:02 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:02.040 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:02 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:02.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:02 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:02.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:02 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:02.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:02 ntpd[17095]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:02.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:02 ntpd[17095]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:03.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:03 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:03.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:03 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:03.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:03 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:03.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:03 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:03.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:03 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:03.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:03 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:03.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:03 ntpd[17095]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:03.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:03 ntpd[17095]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:03.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:03 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:03.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:03 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:03.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:03 ntpd[17095]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:03.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:03 ntpd[17095]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:03.044 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:03 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:03.044 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:03 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:03.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:03 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:03.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:03 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:03.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:03 ntpd[17144]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:03.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:03 ntpd[17144]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:03.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:03 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:03.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:03 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:03.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:03 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:03.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:03 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:03.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:03 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:03.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:03 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:03.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:03 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:03.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:03 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:03.051 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:03 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:03.051 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:03 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:03.051 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:03 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:03.051 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:03 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:03.073 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:03 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:03.073 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:03 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:03.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:03 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:03.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:03 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:03.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:03 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:03.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:03 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:03.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:03 ntpd[17157]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:03.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:03 ntpd[17157]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:03.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:03 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:03.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:03 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:03.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:03 ntpd[17157]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:03.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:03 ntpd[17157]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:04.040 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:04 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:04.040 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:04 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:04.040 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:04 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:04.040 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:04 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:04.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:04 ntpd[17095]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:04.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:04 ntpd[17095]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:04.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:04 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:04.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:04 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:04.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:04 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:04.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:04 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:04.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:04 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:04.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:04 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:04.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:04 ntpd[17144]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:04.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:04 ntpd[17144]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:04.049 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:04 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:04.049 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:04 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:04.049 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:04 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:04.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:04 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:04.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:04 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:04.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:04 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:05.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:05 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:05 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:05 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:05 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:05 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:05 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:05 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.042 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:05 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.042 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:05 ntpd[17095]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:05.042 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:05 ntpd[17095]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:05.042 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:05 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.042 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:05 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.042 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:05 ntpd[17095]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:05.042 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:05 ntpd[17095]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:05.044 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:05 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:05 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:05 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:05 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:05 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:05 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:05 ntpd[17144]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:05 ntpd[17144]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:05 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:05 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:05 ntpd[17144]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:05 ntpd[17144]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:05 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:05 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:05 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:05 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:05 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:05 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:05 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:05 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:05 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:05 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:05 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:05 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:05.075 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.075 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.075 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.075 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.075 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.075 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.075 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.075 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.075 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:05.075 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:05.076 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.076 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.076 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:05.076 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:05.076 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:05.076 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:05.076 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:05.076 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:05 ntpd[17157]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:06.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:06 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:06.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:06 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:06.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:06 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:06.042 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:06 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:06.042 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:06 ntpd[17095]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:06.042 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:06 ntpd[17095]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:06.047 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:06 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:06.047 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:06 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:06.047 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:06 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:06.047 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:06 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:06.048 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:06 ntpd[17144]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:06.048 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:06 ntpd[17144]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:06.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:06 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:06.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:06 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:06.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:06 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:06.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:06 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:06.051 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:06 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:06.051 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:06 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:07.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:07 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:07 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:07 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:07 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:07 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:07 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:07 ntpd[17095]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:07.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:07 ntpd[17095]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:07.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:07 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:07 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.042 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:07 ntpd[17095]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:07.042 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:07 ntpd[17095]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:07.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:07 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:07 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:07 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:07 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:07 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:07 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.045 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:07 ntpd[17144]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:07.046 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:07 ntpd[17144]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:07.046 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:07 ntpd[17144]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.046 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:07 ntpd[17144]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.046 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:07 ntpd[17144]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:07.046 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:07 ntpd[17144]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:07.046 INFO:teuthology.orchestra.run.smithi012.stdout:26 Apr 01:04:07 ntpd[17144]: ntpd: time slew -0.000203 s 2024-04-26T01:04:07.046 INFO:teuthology.orchestra.run.smithi012.stdout:ntpd: time slew -0.000203s 2024-04-26T01:04:07.047 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:07 ntpd[17144]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T01:04:07.047 INFO:teuthology.orchestra.run.smithi012.stderr:26 Apr 01:04:07 ntpd[17144]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T01:04:07.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:07 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:07 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:07 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:07 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:07 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:07 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:07 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:07.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:07 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:07.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:07 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:07 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.050 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:07 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:07.051 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:07 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:07.051 INFO:teuthology.orchestra.run.smithi116.stdout:26 Apr 01:04:07 ntpd[17205]: ntpd: time slew -0.002769 s 2024-04-26T01:04:07.051 INFO:teuthology.orchestra.run.smithi116.stdout:ntpd: time slew -0.002769s 2024-04-26T01:04:07.051 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:07 ntpd[17205]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T01:04:07.051 INFO:teuthology.orchestra.run.smithi116.stderr:26 Apr 01:04:07 ntpd[17205]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T01:04:07.073 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:07 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.073 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:07 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.073 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:07 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.073 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:07 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.073 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:07 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:07 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:07 ntpd[17157]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:07.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:07 ntpd[17157]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:07.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:07 ntpd[17157]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:07.074 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:07 ntpd[17157]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:07.074 INFO:teuthology.orchestra.run.smithi164.stdout:26 Apr 01:04:07 ntpd[17157]: ntpd: time slew +0.000369 s 2024-04-26T01:04:07.074 INFO:teuthology.orchestra.run.smithi164.stdout:ntpd: time slew +0.000369s 2024-04-26T01:04:07.075 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:07 ntpd[17157]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T01:04:07.075 INFO:teuthology.orchestra.run.smithi164.stderr:26 Apr 01:04:07 ntpd[17157]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T01:04:07.097 INFO:teuthology.orchestra.run.smithi012.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:04:07.098 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================== 2024-04-26T01:04:07.098 INFO:teuthology.orchestra.run.smithi012.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:07.098 INFO:teuthology.orchestra.run.smithi012.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:07.098 INFO:teuthology.orchestra.run.smithi012.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:07.098 INFO:teuthology.orchestra.run.smithi012.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:04:07.109 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================== 2024-04-26T01:04:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:07.110 INFO:teuthology.orchestra.run.smithi116.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:04:07.124 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================== 2024-04-26T01:04:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:08.040 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:08 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:08.040 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:08 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:08.040 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:08 ntpd[17095]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:04:08.040 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:08 ntpd[17095]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:04:08.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:08 ntpd[17095]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:04:08.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:08 ntpd[17095]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:04:08.041 INFO:teuthology.orchestra.run.smithi008.stdout:26 Apr 01:04:08 ntpd[17095]: ntpd: time slew -0.000518 s 2024-04-26T01:04:08.041 INFO:teuthology.orchestra.run.smithi008.stdout:ntpd: time slew -0.000518s 2024-04-26T01:04:08.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:08 ntpd[17095]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T01:04:08.041 INFO:teuthology.orchestra.run.smithi008.stderr:26 Apr 01:04:08 ntpd[17095]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T01:04:08.090 INFO:teuthology.orchestra.run.smithi008.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:04:08.090 INFO:teuthology.orchestra.run.smithi008.stdout:============================================================================== 2024-04-26T01:04:08.090 INFO:teuthology.orchestra.run.smithi008.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:08.090 INFO:teuthology.orchestra.run.smithi008.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:08.090 INFO:teuthology.orchestra.run.smithi008.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:08.090 INFO:teuthology.orchestra.run.smithi008.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:04:08.092 INFO:teuthology.run_tasks:Running task install... 2024-04-26T01:04:08.103 DEBUG:teuthology.task.install:project ceph 2024-04-26T01:04:08.103 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-26T01:04:08.103 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-26T01:04:08.103 INFO:teuthology.task.install:Using flavor: default 2024-04-26T01:04:08.113 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T01:04:08.113 INFO:teuthology.task.install:extra packages: [] 2024-04-26T01:04:08.114 DEBUG:teuthology.orchestra.run.smithi008:> sudo apt-key list | grep Ceph 2024-04-26T01:04:08.114 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-key list | grep Ceph 2024-04-26T01:04:08.115 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-key list | grep Ceph 2024-04-26T01:04:08.116 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-key list | grep Ceph 2024-04-26T01:04:08.174 INFO:teuthology.orchestra.run.smithi116.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T01:04:08.181 INFO:teuthology.orchestra.run.smithi164.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T01:04:08.186 INFO:teuthology.orchestra.run.smithi008.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T01:04:08.187 INFO:teuthology.orchestra.run.smithi012.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T01:04:08.201 INFO:teuthology.orchestra.run.smithi116.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T01:04:08.201 INFO:teuthology.orchestra.run.smithi116.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T01:04:08.202 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T01:04:08.203 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:04:08.211 INFO:teuthology.orchestra.run.smithi164.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T01:04:08.212 INFO:teuthology.orchestra.run.smithi164.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T01:04:08.212 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T01:04:08.213 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:04:08.216 INFO:teuthology.orchestra.run.smithi008.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T01:04:08.216 INFO:teuthology.orchestra.run.smithi008.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T01:04:08.216 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T01:04:08.217 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:04:08.222 INFO:teuthology.orchestra.run.smithi012.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T01:04:08.222 INFO:teuthology.orchestra.run.smithi012.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T01:04:08.222 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T01:04:08.222 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:04:08.356 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default/ 2024-04-26T01:04:08.356 INFO:teuthology.task.install.deb:Package version is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:04:08.395 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default/ 2024-04-26T01:04:08.395 INFO:teuthology.task.install.deb:Package version is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:04:08.403 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default/ 2024-04-26T01:04:08.403 INFO:teuthology.task.install.deb:Package version is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:04:08.467 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:04:08.467 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T01:04:08.481 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-get update 2024-04-26T01:04:08.491 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default/ 2024-04-26T01:04:08.491 INFO:teuthology.task.install.deb:Package version is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:04:08.600 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:04:08.601 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T01:04:08.615 DEBUG:teuthology.orchestra.run.smithi008:> sudo apt-get update 2024-04-26T01:04:08.654 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:04:08.654 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T01:04:08.658 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:04:08.658 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T01:04:08.673 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get update 2024-04-26T01:04:08.700 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:04:08.700 INFO:teuthology.orchestra.run.smithi164.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T01:04:08.704 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get update 2024-04-26T01:04:08.704 INFO:teuthology.orchestra.run.smithi164.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T01:04:08.725 INFO:teuthology.orchestra.run.smithi164.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T01:04:08.745 INFO:teuthology.orchestra.run.smithi164.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T01:04:08.747 INFO:teuthology.orchestra.run.smithi164.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T01:04:08.768 INFO:teuthology.orchestra.run.smithi164.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-26T01:04:08.797 INFO:teuthology.orchestra.run.smithi164.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-26T01:04:08.841 INFO:teuthology.orchestra.run.smithi008.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T01:04:08.851 INFO:teuthology.orchestra.run.smithi008.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T01:04:08.862 INFO:teuthology.orchestra.run.smithi008.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T01:04:08.885 INFO:teuthology.orchestra.run.smithi008.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T01:04:08.894 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:04:08.894 INFO:teuthology.orchestra.run.smithi012.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T01:04:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T01:04:08.903 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:04:08.903 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T01:04:08.905 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T01:04:08.906 INFO:teuthology.orchestra.run.smithi008.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-26T01:04:08.914 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T01:04:08.915 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T01:04:08.915 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:04:08.919 INFO:teuthology.orchestra.run.smithi012.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T01:04:08.925 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T01:04:08.935 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T01:04:08.941 INFO:teuthology.orchestra.run.smithi012.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T01:04:08.944 INFO:teuthology.orchestra.run.smithi008.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-26T01:04:08.962 INFO:teuthology.orchestra.run.smithi012.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-26T01:04:08.988 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T01:04:08.990 INFO:teuthology.orchestra.run.smithi012.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-26T01:04:09.007 INFO:teuthology.orchestra.run.smithi116.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T01:04:09.043 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T01:04:09.069 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T01:04:09.081 INFO:teuthology.orchestra.run.smithi116.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T01:04:09.150 INFO:teuthology.orchestra.run.smithi164.stdout:Fetched 258 kB in 0s (529 kB/s) 2024-04-26T01:04:09.261 INFO:teuthology.orchestra.run.smithi012.stdout:Fetched 139 kB in 0s (341 kB/s) 2024-04-26T01:04:09.297 INFO:teuthology.orchestra.run.smithi008.stdout:Fetched 139 kB in 0s (281 kB/s) 2024-04-26T01:04:09.367 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-26T01:04:09.419 INFO:teuthology.orchestra.run.smithi116.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-26T01:04:09.501 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 28.9 kB in 1s (47.4 kB/s) 2024-04-26T01:04:10.155 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2024-04-26T01:04:10.171 DEBUG:teuthology.orchestra.run.smithi164:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2445-gd2c45d97-1jammy cephadm=19.0.0-2445-gd2c45d97-1jammy ceph-mds=19.0.0-2445-gd2c45d97-1jammy ceph-mgr=19.0.0-2445-gd2c45d97-1jammy ceph-common=19.0.0-2445-gd2c45d97-1jammy ceph-fuse=19.0.0-2445-gd2c45d97-1jammy ceph-test=19.0.0-2445-gd2c45d97-1jammy ceph-volume=19.0.0-2445-gd2c45d97-1jammy radosgw=19.0.0-2445-gd2c45d97-1jammy python3-rados=19.0.0-2445-gd2c45d97-1jammy python3-rgw=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy python3-rbd=19.0.0-2445-gd2c45d97-1jammy libcephfs2=19.0.0-2445-gd2c45d97-1jammy libcephfs-dev=19.0.0-2445-gd2c45d97-1jammy librados2=19.0.0-2445-gd2c45d97-1jammy librbd1=19.0.0-2445-gd2c45d97-1jammy rbd-fuse=19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:04:10.232 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2024-04-26T01:04:10.239 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-04-26T01:04:10.257 DEBUG:teuthology.orchestra.run.smithi012:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2445-gd2c45d97-1jammy cephadm=19.0.0-2445-gd2c45d97-1jammy ceph-mds=19.0.0-2445-gd2c45d97-1jammy ceph-mgr=19.0.0-2445-gd2c45d97-1jammy ceph-common=19.0.0-2445-gd2c45d97-1jammy ceph-fuse=19.0.0-2445-gd2c45d97-1jammy ceph-test=19.0.0-2445-gd2c45d97-1jammy ceph-volume=19.0.0-2445-gd2c45d97-1jammy radosgw=19.0.0-2445-gd2c45d97-1jammy python3-rados=19.0.0-2445-gd2c45d97-1jammy python3-rgw=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy python3-rbd=19.0.0-2445-gd2c45d97-1jammy libcephfs2=19.0.0-2445-gd2c45d97-1jammy libcephfs-dev=19.0.0-2445-gd2c45d97-1jammy librados2=19.0.0-2445-gd2c45d97-1jammy librbd1=19.0.0-2445-gd2c45d97-1jammy rbd-fuse=19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:04:10.292 INFO:teuthology.orchestra.run.smithi008.stdout:Reading package lists... 2024-04-26T01:04:10.307 DEBUG:teuthology.orchestra.run.smithi008:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2445-gd2c45d97-1jammy cephadm=19.0.0-2445-gd2c45d97-1jammy ceph-mds=19.0.0-2445-gd2c45d97-1jammy ceph-mgr=19.0.0-2445-gd2c45d97-1jammy ceph-common=19.0.0-2445-gd2c45d97-1jammy ceph-fuse=19.0.0-2445-gd2c45d97-1jammy ceph-test=19.0.0-2445-gd2c45d97-1jammy ceph-volume=19.0.0-2445-gd2c45d97-1jammy radosgw=19.0.0-2445-gd2c45d97-1jammy python3-rados=19.0.0-2445-gd2c45d97-1jammy python3-rgw=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy python3-rbd=19.0.0-2445-gd2c45d97-1jammy libcephfs2=19.0.0-2445-gd2c45d97-1jammy libcephfs-dev=19.0.0-2445-gd2c45d97-1jammy librados2=19.0.0-2445-gd2c45d97-1jammy librbd1=19.0.0-2445-gd2c45d97-1jammy rbd-fuse=19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:04:10.322 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-04-26T01:04:10.360 INFO:teuthology.orchestra.run.smithi008.stdout:Reading package lists... 2024-04-26T01:04:10.441 INFO:teuthology.orchestra.run.smithi164.stdout:Building dependency tree... 2024-04-26T01:04:10.442 INFO:teuthology.orchestra.run.smithi164.stdout:Reading state information... 2024-04-26T01:04:10.470 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-04-26T01:04:10.487 DEBUG:teuthology.orchestra.run.smithi116:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2445-gd2c45d97-1jammy cephadm=19.0.0-2445-gd2c45d97-1jammy ceph-mds=19.0.0-2445-gd2c45d97-1jammy ceph-mgr=19.0.0-2445-gd2c45d97-1jammy ceph-common=19.0.0-2445-gd2c45d97-1jammy ceph-fuse=19.0.0-2445-gd2c45d97-1jammy ceph-test=19.0.0-2445-gd2c45d97-1jammy ceph-volume=19.0.0-2445-gd2c45d97-1jammy radosgw=19.0.0-2445-gd2c45d97-1jammy python3-rados=19.0.0-2445-gd2c45d97-1jammy python3-rgw=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy python3-rbd=19.0.0-2445-gd2c45d97-1jammy libcephfs2=19.0.0-2445-gd2c45d97-1jammy libcephfs-dev=19.0.0-2445-gd2c45d97-1jammy librados2=19.0.0-2445-gd2c45d97-1jammy librbd1=19.0.0-2445-gd2c45d97-1jammy rbd-fuse=19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:04:10.525 INFO:teuthology.orchestra.run.smithi012.stdout:Building dependency tree... 2024-04-26T01:04:10.526 INFO:teuthology.orchestra.run.smithi012.stdout:Reading state information... 2024-04-26T01:04:10.540 INFO:teuthology.orchestra.run.smithi008.stdout:Building dependency tree... 2024-04-26T01:04:10.541 INFO:teuthology.orchestra.run.smithi008.stdout:Reading state information... 2024-04-26T01:04:10.550 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-04-26T01:04:10.646 INFO:teuthology.orchestra.run.smithi164.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:04:10.646 INFO:teuthology.orchestra.run.smithi164.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:04:10.646 INFO:teuthology.orchestra.run.smithi164.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:04:10.646 INFO:teuthology.orchestra.run.smithi164.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:04:10.647 INFO:teuthology.orchestra.run.smithi164.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T01:04:10.647 INFO:teuthology.orchestra.run.smithi164.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T01:04:10.647 INFO:teuthology.orchestra.run.smithi164.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:04:10.648 INFO:teuthology.orchestra.run.smithi164.stdout:The following additional packages will be installed: 2024-04-26T01:04:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T01:04:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T01:04:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T01:04:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T01:04:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T01:04:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T01:04:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T01:04:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T01:04:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T01:04:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:04:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:04:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:04:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:04:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:04:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:04:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:04:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T01:04:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T01:04:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T01:04:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T01:04:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T01:04:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T01:04:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T01:04:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet zip 2024-04-26T01:04:10.651 INFO:teuthology.orchestra.run.smithi164.stdout:Suggested packages: 2024-04-26T01:04:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T01:04:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T01:04:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T01:04:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T01:04:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T01:04:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T01:04:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T01:04:10.652 INFO:teuthology.orchestra.run.smithi164.stdout:Recommended packages: 2024-04-26T01:04:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: btrfs-tools 2024-04-26T01:04:10.721 INFO:teuthology.orchestra.run.smithi008.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:04:10.722 INFO:teuthology.orchestra.run.smithi008.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:04:10.722 INFO:teuthology.orchestra.run.smithi008.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:04:10.722 INFO:teuthology.orchestra.run.smithi008.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:04:10.722 INFO:teuthology.orchestra.run.smithi008.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T01:04:10.722 INFO:teuthology.orchestra.run.smithi008.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T01:04:10.723 INFO:teuthology.orchestra.run.smithi008.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:04:10.723 INFO:teuthology.orchestra.run.smithi008.stdout:The following additional packages will be installed: 2024-04-26T01:04:10.723 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T01:04:10.723 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T01:04:10.724 INFO:teuthology.orchestra.run.smithi008.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T01:04:10.724 INFO:teuthology.orchestra.run.smithi008.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T01:04:10.724 INFO:teuthology.orchestra.run.smithi008.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T01:04:10.725 INFO:teuthology.orchestra.run.smithi008.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T01:04:10.725 INFO:teuthology.orchestra.run.smithi008.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T01:04:10.725 INFO:teuthology.orchestra.run.smithi008.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T01:04:10.725 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T01:04:10.725 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:04:10.725 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:04:10.725 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:04:10.725 INFO:teuthology.orchestra.run.smithi008.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:04:10.725 INFO:teuthology.orchestra.run.smithi008.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:04:10.725 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:04:10.725 INFO:teuthology.orchestra.run.smithi008.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:04:10.725 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T01:04:10.725 INFO:teuthology.orchestra.run.smithi008.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T01:04:10.726 INFO:teuthology.orchestra.run.smithi008.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T01:04:10.726 INFO:teuthology.orchestra.run.smithi008.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T01:04:10.726 INFO:teuthology.orchestra.run.smithi008.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T01:04:10.726 INFO:teuthology.orchestra.run.smithi008.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T01:04:10.726 INFO:teuthology.orchestra.run.smithi008.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T01:04:10.726 INFO:teuthology.orchestra.run.smithi008.stdout: xmlstarlet zip 2024-04-26T01:04:10.726 INFO:teuthology.orchestra.run.smithi008.stdout:Suggested packages: 2024-04-26T01:04:10.726 INFO:teuthology.orchestra.run.smithi008.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T01:04:10.726 INFO:teuthology.orchestra.run.smithi008.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T01:04:10.727 INFO:teuthology.orchestra.run.smithi008.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T01:04:10.727 INFO:teuthology.orchestra.run.smithi008.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T01:04:10.727 INFO:teuthology.orchestra.run.smithi008.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T01:04:10.727 INFO:teuthology.orchestra.run.smithi008.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T01:04:10.727 INFO:teuthology.orchestra.run.smithi008.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T01:04:10.727 INFO:teuthology.orchestra.run.smithi008.stdout:Recommended packages: 2024-04-26T01:04:10.727 INFO:teuthology.orchestra.run.smithi008.stdout: btrfs-tools 2024-04-26T01:04:10.731 INFO:teuthology.orchestra.run.smithi012.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:04:10.731 INFO:teuthology.orchestra.run.smithi012.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:04:10.731 INFO:teuthology.orchestra.run.smithi012.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:04:10.732 INFO:teuthology.orchestra.run.smithi012.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:04:10.732 INFO:teuthology.orchestra.run.smithi012.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T01:04:10.732 INFO:teuthology.orchestra.run.smithi012.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T01:04:10.732 INFO:teuthology.orchestra.run.smithi012.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:04:10.733 INFO:teuthology.orchestra.run.smithi012.stdout:The following additional packages will be installed: 2024-04-26T01:04:10.733 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T01:04:10.735 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T01:04:10.735 INFO:teuthology.orchestra.run.smithi012.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T01:04:10.735 INFO:teuthology.orchestra.run.smithi012.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T01:04:10.735 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T01:04:10.735 INFO:teuthology.orchestra.run.smithi012.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T01:04:10.735 INFO:teuthology.orchestra.run.smithi012.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T01:04:10.735 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T01:04:10.735 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T01:04:10.735 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:04:10.736 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:04:10.736 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:04:10.736 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:04:10.736 INFO:teuthology.orchestra.run.smithi012.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:04:10.736 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:04:10.736 INFO:teuthology.orchestra.run.smithi012.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:04:10.736 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T01:04:10.736 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T01:04:10.736 INFO:teuthology.orchestra.run.smithi012.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T01:04:10.736 INFO:teuthology.orchestra.run.smithi012.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T01:04:10.736 INFO:teuthology.orchestra.run.smithi012.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T01:04:10.736 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T01:04:10.736 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T01:04:10.737 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet zip 2024-04-26T01:04:10.737 INFO:teuthology.orchestra.run.smithi012.stdout:Suggested packages: 2024-04-26T01:04:10.737 INFO:teuthology.orchestra.run.smithi012.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T01:04:10.737 INFO:teuthology.orchestra.run.smithi012.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T01:04:10.737 INFO:teuthology.orchestra.run.smithi012.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T01:04:10.737 INFO:teuthology.orchestra.run.smithi012.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T01:04:10.737 INFO:teuthology.orchestra.run.smithi012.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T01:04:10.737 INFO:teuthology.orchestra.run.smithi012.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T01:04:10.737 INFO:teuthology.orchestra.run.smithi012.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T01:04:10.737 INFO:teuthology.orchestra.run.smithi012.stdout:Recommended packages: 2024-04-26T01:04:10.738 INFO:teuthology.orchestra.run.smithi012.stdout: btrfs-tools 2024-04-26T01:04:10.750 INFO:teuthology.orchestra.run.smithi164.stdout:The following NEW packages will be installed: 2024-04-26T01:04:10.750 INFO:teuthology.orchestra.run.smithi164.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T01:04:10.750 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T01:04:10.750 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T01:04:10.750 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T01:04:10.750 INFO:teuthology.orchestra.run.smithi164.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T01:04:10.750 INFO:teuthology.orchestra.run.smithi164.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T01:04:10.754 INFO:teuthology.orchestra.run.smithi116.stdout:Building dependency tree... 2024-04-26T01:04:10.754 INFO:teuthology.orchestra.run.smithi116.stdout:Reading state information... 2024-04-26T01:04:10.755 INFO:teuthology.orchestra.run.smithi164.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T01:04:10.756 INFO:teuthology.orchestra.run.smithi164.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T01:04:10.756 INFO:teuthology.orchestra.run.smithi164.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T01:04:10.756 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T01:04:10.756 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T01:04:10.756 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:04:10.756 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:04:10.756 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:04:10.756 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:04:10.756 INFO:teuthology.orchestra.run.smithi164.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:04:10.756 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:04:10.756 INFO:teuthology.orchestra.run.smithi164.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:04:10.756 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T01:04:10.756 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T01:04:10.756 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T01:04:10.757 INFO:teuthology.orchestra.run.smithi164.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T01:04:10.757 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T01:04:10.757 INFO:teuthology.orchestra.run.smithi164.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T01:04:10.757 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T01:04:10.757 INFO:teuthology.orchestra.run.smithi164.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T01:04:10.821 INFO:teuthology.orchestra.run.smithi164.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T01:04:10.821 INFO:teuthology.orchestra.run.smithi164.stdout:Need to get 179 MB of archives. 2024-04-26T01:04:10.821 INFO:teuthology.orchestra.run.smithi164.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-26T01:04:10.821 INFO:teuthology.orchestra.run.smithi164.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T01:04:10.827 INFO:teuthology.orchestra.run.smithi008.stdout:The following NEW packages will be installed: 2024-04-26T01:04:10.827 INFO:teuthology.orchestra.run.smithi008.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T01:04:10.827 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T01:04:10.829 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T01:04:10.829 INFO:teuthology.orchestra.run.smithi008.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T01:04:10.829 INFO:teuthology.orchestra.run.smithi008.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T01:04:10.829 INFO:teuthology.orchestra.run.smithi008.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T01:04:10.829 INFO:teuthology.orchestra.run.smithi008.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T01:04:10.829 INFO:teuthology.orchestra.run.smithi008.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T01:04:10.829 INFO:teuthology.orchestra.run.smithi008.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T01:04:10.829 INFO:teuthology.orchestra.run.smithi008.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T01:04:10.830 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T01:04:10.830 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:04:10.830 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:04:10.830 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:04:10.830 INFO:teuthology.orchestra.run.smithi008.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:04:10.830 INFO:teuthology.orchestra.run.smithi008.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:04:10.830 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:04:10.830 INFO:teuthology.orchestra.run.smithi008.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:04:10.830 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T01:04:10.830 INFO:teuthology.orchestra.run.smithi008.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T01:04:10.830 INFO:teuthology.orchestra.run.smithi008.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T01:04:10.830 INFO:teuthology.orchestra.run.smithi008.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T01:04:10.830 INFO:teuthology.orchestra.run.smithi008.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T01:04:10.831 INFO:teuthology.orchestra.run.smithi008.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T01:04:10.831 INFO:teuthology.orchestra.run.smithi008.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T01:04:10.831 INFO:teuthology.orchestra.run.smithi008.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T01:04:10.845 INFO:teuthology.orchestra.run.smithi012.stdout:The following NEW packages will be installed: 2024-04-26T01:04:10.845 INFO:teuthology.orchestra.run.smithi012.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T01:04:10.845 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T01:04:10.845 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T01:04:10.845 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T01:04:10.846 INFO:teuthology.orchestra.run.smithi012.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T01:04:10.846 INFO:teuthology.orchestra.run.smithi012.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T01:04:10.846 INFO:teuthology.orchestra.run.smithi012.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T01:04:10.846 INFO:teuthology.orchestra.run.smithi012.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T01:04:10.846 INFO:teuthology.orchestra.run.smithi012.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T01:04:10.846 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T01:04:10.846 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T01:04:10.846 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:04:10.846 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:04:10.846 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:04:10.846 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:04:10.847 INFO:teuthology.orchestra.run.smithi012.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:04:10.847 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:04:10.847 INFO:teuthology.orchestra.run.smithi012.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:04:10.847 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T01:04:10.847 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T01:04:10.847 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T01:04:10.847 INFO:teuthology.orchestra.run.smithi012.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T01:04:10.847 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T01:04:10.847 INFO:teuthology.orchestra.run.smithi012.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T01:04:10.847 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T01:04:10.847 INFO:teuthology.orchestra.run.smithi012.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T01:04:10.892 INFO:teuthology.orchestra.run.smithi164.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2445-gd2c45d97-1jammy [3,541 kB] 2024-04-26T01:04:10.940 INFO:teuthology.orchestra.run.smithi164.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T01:04:10.946 INFO:teuthology.orchestra.run.smithi164.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T01:04:10.947 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:04:10.947 INFO:teuthology.orchestra.run.smithi116.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:04:10.947 INFO:teuthology.orchestra.run.smithi116.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:04:10.948 INFO:teuthology.orchestra.run.smithi116.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:04:10.948 INFO:teuthology.orchestra.run.smithi116.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T01:04:10.948 INFO:teuthology.orchestra.run.smithi116.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T01:04:10.949 INFO:teuthology.orchestra.run.smithi116.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:04:10.949 INFO:teuthology.orchestra.run.smithi116.stdout:The following additional packages will be installed: 2024-04-26T01:04:10.949 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T01:04:10.949 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T01:04:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T01:04:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T01:04:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T01:04:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T01:04:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T01:04:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T01:04:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T01:04:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:04:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:04:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:04:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:04:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:04:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:04:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:04:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T01:04:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T01:04:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T01:04:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T01:04:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T01:04:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T01:04:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T01:04:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet zip 2024-04-26T01:04:10.952 INFO:teuthology.orchestra.run.smithi116.stdout:Suggested packages: 2024-04-26T01:04:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T01:04:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T01:04:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T01:04:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T01:04:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T01:04:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T01:04:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T01:04:10.953 INFO:teuthology.orchestra.run.smithi116.stdout:Recommended packages: 2024-04-26T01:04:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: btrfs-tools 2024-04-26T01:04:10.975 INFO:teuthology.orchestra.run.smithi164.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T01:04:10.990 INFO:teuthology.orchestra.run.smithi012.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T01:04:10.990 INFO:teuthology.orchestra.run.smithi012.stdout:Need to get 179 MB of archives. 2024-04-26T01:04:10.990 INFO:teuthology.orchestra.run.smithi012.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-26T01:04:10.990 INFO:teuthology.orchestra.run.smithi012.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2445-gd2c45d97-1jammy [3,541 kB] 2024-04-26T01:04:11.020 INFO:teuthology.orchestra.run.smithi008.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T01:04:11.020 INFO:teuthology.orchestra.run.smithi008.stdout:Need to get 179 MB of archives. 2024-04-26T01:04:11.021 INFO:teuthology.orchestra.run.smithi008.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-26T01:04:11.021 INFO:teuthology.orchestra.run.smithi008.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T01:04:11.035 INFO:teuthology.orchestra.run.smithi012.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T01:04:11.057 INFO:teuthology.orchestra.run.smithi116.stdout:The following NEW packages will be installed: 2024-04-26T01:04:11.057 INFO:teuthology.orchestra.run.smithi116.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T01:04:11.057 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T01:04:11.057 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T01:04:11.058 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T01:04:11.058 INFO:teuthology.orchestra.run.smithi116.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T01:04:11.058 INFO:teuthology.orchestra.run.smithi116.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T01:04:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T01:04:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T01:04:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T01:04:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T01:04:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T01:04:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:04:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:04:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:04:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:04:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:04:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:04:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:04:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T01:04:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T01:04:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T01:04:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T01:04:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T01:04:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T01:04:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T01:04:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T01:04:11.069 INFO:teuthology.orchestra.run.smithi164.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T01:04:11.075 INFO:teuthology.orchestra.run.smithi164.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T01:04:11.097 INFO:teuthology.orchestra.run.smithi008.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2445-gd2c45d97-1jammy [3,541 kB] 2024-04-26T01:04:11.101 INFO:teuthology.orchestra.run.smithi164.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T01:04:11.106 INFO:teuthology.orchestra.run.smithi164.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T01:04:11.107 INFO:teuthology.orchestra.run.smithi164.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T01:04:11.109 INFO:teuthology.orchestra.run.smithi164.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T01:04:11.122 INFO:teuthology.orchestra.run.smithi164.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T01:04:11.125 INFO:teuthology.orchestra.run.smithi164.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T01:04:11.129 INFO:teuthology.orchestra.run.smithi164.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T01:04:11.131 INFO:teuthology.orchestra.run.smithi164.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T01:04:11.175 INFO:teuthology.orchestra.run.smithi164.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T01:04:11.179 INFO:teuthology.orchestra.run.smithi164.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T01:04:11.183 INFO:teuthology.orchestra.run.smithi164.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T01:04:11.186 INFO:teuthology.orchestra.run.smithi164.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T01:04:11.186 INFO:teuthology.orchestra.run.smithi164.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T01:04:11.187 INFO:teuthology.orchestra.run.smithi164.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T01:04:11.212 INFO:teuthology.orchestra.run.smithi164.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T01:04:11.212 INFO:teuthology.orchestra.run.smithi164.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T01:04:11.213 INFO:teuthology.orchestra.run.smithi164.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T01:04:11.214 INFO:teuthology.orchestra.run.smithi164.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T01:04:11.235 INFO:teuthology.orchestra.run.smithi164.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T01:04:11.236 INFO:teuthology.orchestra.run.smithi164.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T01:04:11.237 INFO:teuthology.orchestra.run.smithi164.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T01:04:11.241 INFO:teuthology.orchestra.run.smithi164.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T01:04:11.242 INFO:teuthology.orchestra.run.smithi164.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T01:04:11.243 INFO:teuthology.orchestra.run.smithi164.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T01:04:11.248 INFO:teuthology.orchestra.run.smithi116.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T01:04:11.248 INFO:teuthology.orchestra.run.smithi116.stdout:Need to get 179 MB of archives. 2024-04-26T01:04:11.248 INFO:teuthology.orchestra.run.smithi116.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-26T01:04:11.248 INFO:teuthology.orchestra.run.smithi116.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T01:04:11.258 INFO:teuthology.orchestra.run.smithi164.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T01:04:11.259 INFO:teuthology.orchestra.run.smithi164.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T01:04:11.260 INFO:teuthology.orchestra.run.smithi164.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T01:04:11.261 INFO:teuthology.orchestra.run.smithi164.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T01:04:11.281 INFO:teuthology.orchestra.run.smithi164.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T01:04:11.283 INFO:teuthology.orchestra.run.smithi164.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T01:04:11.283 INFO:teuthology.orchestra.run.smithi164.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T01:04:11.293 INFO:teuthology.orchestra.run.smithi164.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T01:04:11.294 INFO:teuthology.orchestra.run.smithi164.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T01:04:11.295 INFO:teuthology.orchestra.run.smithi164.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T01:04:11.304 INFO:teuthology.orchestra.run.smithi164.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T01:04:11.307 INFO:teuthology.orchestra.run.smithi164.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T01:04:11.310 INFO:teuthology.orchestra.run.smithi164.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T01:04:11.311 INFO:teuthology.orchestra.run.smithi164.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T01:04:11.330 INFO:teuthology.orchestra.run.smithi164.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T01:04:11.332 INFO:teuthology.orchestra.run.smithi164.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T01:04:11.336 INFO:teuthology.orchestra.run.smithi164.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T01:04:11.337 INFO:teuthology.orchestra.run.smithi164.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T01:04:11.339 INFO:teuthology.orchestra.run.smithi164.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T01:04:11.343 INFO:teuthology.orchestra.run.smithi164.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T01:04:11.346 INFO:teuthology.orchestra.run.smithi116.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2445-gd2c45d97-1jammy [3,541 kB] 2024-04-26T01:04:11.348 INFO:teuthology.orchestra.run.smithi164.stdout:Get:52 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2445-gd2c45d97-1jammy [3,218 kB] 2024-04-26T01:04:11.451 INFO:teuthology.orchestra.run.smithi012.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2445-gd2c45d97-1jammy [3,218 kB] 2024-04-26T01:04:11.456 INFO:teuthology.orchestra.run.smithi012.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T01:04:11.468 INFO:teuthology.orchestra.run.smithi012.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T01:04:11.471 INFO:teuthology.orchestra.run.smithi164.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T01:04:11.480 INFO:teuthology.orchestra.run.smithi164.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T01:04:11.483 INFO:teuthology.orchestra.run.smithi164.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T01:04:11.483 INFO:teuthology.orchestra.run.smithi164.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T01:04:11.485 INFO:teuthology.orchestra.run.smithi164.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T01:04:11.547 INFO:teuthology.orchestra.run.smithi164.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T01:04:11.548 INFO:teuthology.orchestra.run.smithi008.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T01:04:11.549 INFO:teuthology.orchestra.run.smithi012.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T01:04:11.550 INFO:teuthology.orchestra.run.smithi164.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T01:04:11.551 INFO:teuthology.orchestra.run.smithi164.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T01:04:11.555 INFO:teuthology.orchestra.run.smithi164.stdout:Get:61 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2445-gd2c45d97-1jammy [711 kB] 2024-04-26T01:04:11.575 INFO:teuthology.orchestra.run.smithi164.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T01:04:11.575 INFO:teuthology.orchestra.run.smithi164.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T01:04:11.575 INFO:teuthology.orchestra.run.smithi164.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T01:04:11.576 INFO:teuthology.orchestra.run.smithi164.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T01:04:11.577 INFO:teuthology.orchestra.run.smithi164.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T01:04:11.577 INFO:teuthology.orchestra.run.smithi164.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T01:04:11.585 INFO:teuthology.orchestra.run.smithi008.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T01:04:11.592 INFO:teuthology.orchestra.run.smithi164.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T01:04:11.593 INFO:teuthology.orchestra.run.smithi164.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T01:04:11.596 INFO:teuthology.orchestra.run.smithi164.stdout:Get:70 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2445-gd2c45d97-1jammy [356 kB] 2024-04-26T01:04:11.598 INFO:teuthology.orchestra.run.smithi164.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T01:04:11.615 INFO:teuthology.orchestra.run.smithi164.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T01:04:11.617 INFO:teuthology.orchestra.run.smithi164.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T01:04:11.618 INFO:teuthology.orchestra.run.smithi164.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T01:04:11.622 INFO:teuthology.orchestra.run.smithi164.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T01:04:11.625 INFO:teuthology.orchestra.run.smithi164.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T01:04:11.625 INFO:teuthology.orchestra.run.smithi164.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2445-gd2c45d97-1jammy [32.8 kB] 2024-04-26T01:04:11.626 INFO:teuthology.orchestra.run.smithi164.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T01:04:11.627 INFO:teuthology.orchestra.run.smithi164.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T01:04:11.627 INFO:teuthology.orchestra.run.smithi164.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2445-gd2c45d97-1jammy [180 kB] 2024-04-26T01:04:11.633 INFO:teuthology.orchestra.run.smithi164.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2445-gd2c45d97-1jammy [65.8 kB] 2024-04-26T01:04:11.649 INFO:teuthology.orchestra.run.smithi164.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T01:04:11.650 INFO:teuthology.orchestra.run.smithi164.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2445-gd2c45d97-1jammy [330 kB] 2024-04-26T01:04:11.650 INFO:teuthology.orchestra.run.smithi164.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T01:04:11.654 INFO:teuthology.orchestra.run.smithi164.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T01:04:11.661 INFO:teuthology.orchestra.run.smithi164.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2445-gd2c45d97-1jammy [6,775 kB] 2024-04-26T01:04:11.675 INFO:teuthology.orchestra.run.smithi164.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T01:04:11.676 INFO:teuthology.orchestra.run.smithi164.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T01:04:11.677 INFO:teuthology.orchestra.run.smithi164.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T01:04:11.677 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T01:04:11.679 INFO:teuthology.orchestra.run.smithi164.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T01:04:11.680 INFO:teuthology.orchestra.run.smithi164.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T01:04:11.688 INFO:teuthology.orchestra.run.smithi164.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T01:04:11.688 INFO:teuthology.orchestra.run.smithi164.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T01:04:11.690 INFO:teuthology.orchestra.run.smithi116.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T01:04:11.698 INFO:teuthology.orchestra.run.smithi164.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T01:04:11.702 INFO:teuthology.orchestra.run.smithi164.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T01:04:11.703 INFO:teuthology.orchestra.run.smithi164.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T01:04:11.708 INFO:teuthology.orchestra.run.smithi012.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2445-gd2c45d97-1jammy [711 kB] 2024-04-26T01:04:11.754 INFO:teuthology.orchestra.run.smithi012.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2445-gd2c45d97-1jammy [356 kB] 2024-04-26T01:04:11.774 INFO:teuthology.orchestra.run.smithi116.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T01:04:11.779 INFO:teuthology.orchestra.run.smithi012.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T01:04:11.788 INFO:teuthology.orchestra.run.smithi012.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2445-gd2c45d97-1jammy [32.8 kB] 2024-04-26T01:04:11.790 INFO:teuthology.orchestra.run.smithi012.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2445-gd2c45d97-1jammy [180 kB] 2024-04-26T01:04:11.791 INFO:teuthology.orchestra.run.smithi012.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T01:04:11.820 INFO:teuthology.orchestra.run.smithi012.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2445-gd2c45d97-1jammy [65.8 kB] 2024-04-26T01:04:11.821 INFO:teuthology.orchestra.run.smithi012.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T01:04:11.822 INFO:teuthology.orchestra.run.smithi012.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2445-gd2c45d97-1jammy [330 kB] 2024-04-26T01:04:11.830 INFO:teuthology.orchestra.run.smithi012.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T01:04:11.831 INFO:teuthology.orchestra.run.smithi012.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T01:04:11.832 INFO:teuthology.orchestra.run.smithi012.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T01:04:11.836 INFO:teuthology.orchestra.run.smithi012.stdout:Get:19 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2445-gd2c45d97-1jammy [6,775 kB] 2024-04-26T01:04:11.851 INFO:teuthology.orchestra.run.smithi012.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T01:04:11.981 INFO:teuthology.orchestra.run.smithi008.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T01:04:12.005 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T01:04:12.016 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T01:04:12.019 INFO:teuthology.orchestra.run.smithi012.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T01:04:12.046 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T01:04:12.055 INFO:teuthology.orchestra.run.smithi116.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T01:04:12.056 INFO:teuthology.orchestra.run.smithi116.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T01:04:12.057 INFO:teuthology.orchestra.run.smithi116.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T01:04:12.078 INFO:teuthology.orchestra.run.smithi116.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T01:04:12.159 INFO:teuthology.orchestra.run.smithi164.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2445-gd2c45d97-1jammy [112 kB] 2024-04-26T01:04:12.161 INFO:teuthology.orchestra.run.smithi164.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2445-gd2c45d97-1jammy [467 kB] 2024-04-26T01:04:12.171 INFO:teuthology.orchestra.run.smithi164.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2445-gd2c45d97-1jammy [25.6 MB] 2024-04-26T01:04:12.260 INFO:teuthology.orchestra.run.smithi116.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T01:04:12.421 INFO:teuthology.orchestra.run.smithi012.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T01:04:12.436 INFO:teuthology.orchestra.run.smithi012.stdout:Get:23 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2445-gd2c45d97-1jammy [112 kB] 2024-04-26T01:04:12.445 INFO:teuthology.orchestra.run.smithi012.stdout:Get:24 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2445-gd2c45d97-1jammy [467 kB] 2024-04-26T01:04:12.455 INFO:teuthology.orchestra.run.smithi012.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T01:04:12.457 INFO:teuthology.orchestra.run.smithi012.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T01:04:12.462 INFO:teuthology.orchestra.run.smithi012.stdout:Get:27 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2445-gd2c45d97-1jammy [25.6 MB] 2024-04-26T01:04:12.536 INFO:teuthology.orchestra.run.smithi012.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T01:04:12.587 INFO:teuthology.orchestra.run.smithi012.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T01:04:12.611 INFO:teuthology.orchestra.run.smithi012.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T01:04:12.615 INFO:teuthology.orchestra.run.smithi012.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T01:04:12.616 INFO:teuthology.orchestra.run.smithi012.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T01:04:12.628 INFO:teuthology.orchestra.run.smithi012.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T01:04:12.629 INFO:teuthology.orchestra.run.smithi012.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T01:04:12.732 INFO:teuthology.orchestra.run.smithi008.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2445-gd2c45d97-1jammy [3,218 kB] 2024-04-26T01:04:12.794 INFO:teuthology.orchestra.run.smithi012.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T01:04:12.797 INFO:teuthology.orchestra.run.smithi116.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T01:04:12.877 INFO:teuthology.orchestra.run.smithi012.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T01:04:12.878 INFO:teuthology.orchestra.run.smithi116.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T01:04:12.881 INFO:teuthology.orchestra.run.smithi116.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T01:04:12.906 INFO:teuthology.orchestra.run.smithi012.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T01:04:12.923 INFO:teuthology.orchestra.run.smithi012.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T01:04:12.967 INFO:teuthology.orchestra.run.smithi012.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T01:04:13.182 INFO:teuthology.orchestra.run.smithi012.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T01:04:13.197 INFO:teuthology.orchestra.run.smithi012.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T01:04:13.204 INFO:teuthology.orchestra.run.smithi012.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T01:04:13.209 INFO:teuthology.orchestra.run.smithi012.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T01:04:13.231 INFO:teuthology.orchestra.run.smithi012.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T01:04:13.234 INFO:teuthology.orchestra.run.smithi012.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T01:04:13.354 INFO:teuthology.orchestra.run.smithi116.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T01:04:13.401 INFO:teuthology.orchestra.run.smithi012.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T01:04:13.563 INFO:teuthology.orchestra.run.smithi116.stdout:Get:18 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2445-gd2c45d97-1jammy [3,218 kB] 2024-04-26T01:04:13.732 INFO:teuthology.orchestra.run.smithi012.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T01:04:13.749 INFO:teuthology.orchestra.run.smithi116.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T01:04:13.763 INFO:teuthology.orchestra.run.smithi012.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T01:04:13.773 INFO:teuthology.orchestra.run.smithi012.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T01:04:13.979 INFO:teuthology.orchestra.run.smithi012.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T01:04:13.980 INFO:teuthology.orchestra.run.smithi012.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T01:04:13.984 INFO:teuthology.orchestra.run.smithi012.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T01:04:13.989 INFO:teuthology.orchestra.run.smithi012.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T01:04:14.005 INFO:teuthology.orchestra.run.smithi012.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T01:04:14.018 INFO:teuthology.orchestra.run.smithi164.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2445-gd2c45d97-1jammy [5,084 kB] 2024-04-26T01:04:14.026 INFO:teuthology.orchestra.run.smithi012.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T01:04:14.032 INFO:teuthology.orchestra.run.smithi012.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T01:04:14.061 INFO:teuthology.orchestra.run.smithi116.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T01:04:14.134 INFO:teuthology.orchestra.run.smithi008.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2445-gd2c45d97-1jammy [711 kB] 2024-04-26T01:04:14.164 INFO:teuthology.orchestra.run.smithi116.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T01:04:14.169 INFO:teuthology.orchestra.run.smithi116.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T01:04:14.207 INFO:teuthology.orchestra.run.smithi012.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T01:04:14.303 INFO:teuthology.orchestra.run.smithi164.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2445-gd2c45d97-1jammy [244 kB] 2024-04-26T01:04:14.315 INFO:teuthology.orchestra.run.smithi164.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2445-gd2c45d97-1jammy [124 kB] 2024-04-26T01:04:14.320 INFO:teuthology.orchestra.run.smithi164.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2445-gd2c45d97-1jammy [1,526 kB] 2024-04-26T01:04:14.359 INFO:teuthology.orchestra.run.smithi116.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T01:04:14.385 INFO:teuthology.orchestra.run.smithi116.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T01:04:14.437 INFO:teuthology.orchestra.run.smithi164.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2445-gd2c45d97-1jammy [6,245 kB] 2024-04-26T01:04:14.453 INFO:teuthology.orchestra.run.smithi008.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2445-gd2c45d97-1jammy [356 kB] 2024-04-26T01:04:14.472 INFO:teuthology.orchestra.run.smithi012.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T01:04:14.578 INFO:teuthology.orchestra.run.smithi008.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2445-gd2c45d97-1jammy [32.8 kB] 2024-04-26T01:04:14.588 INFO:teuthology.orchestra.run.smithi008.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2445-gd2c45d97-1jammy [180 kB] 2024-04-26T01:04:14.591 INFO:teuthology.orchestra.run.smithi116.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T01:04:14.669 INFO:teuthology.orchestra.run.smithi008.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2445-gd2c45d97-1jammy [65.8 kB] 2024-04-26T01:04:14.677 INFO:teuthology.orchestra.run.smithi012.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T01:04:14.678 INFO:teuthology.orchestra.run.smithi116.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T01:04:14.678 INFO:teuthology.orchestra.run.smithi012.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T01:04:14.678 INFO:teuthology.orchestra.run.smithi012.stdout:Get:61 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2445-gd2c45d97-1jammy [5,084 kB] 2024-04-26T01:04:14.682 INFO:teuthology.orchestra.run.smithi012.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T01:04:14.684 INFO:teuthology.orchestra.run.smithi008.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2445-gd2c45d97-1jammy [330 kB] 2024-04-26T01:04:14.708 INFO:teuthology.orchestra.run.smithi116.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T01:04:14.726 INFO:teuthology.orchestra.run.smithi116.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T01:04:14.746 INFO:teuthology.orchestra.run.smithi012.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T01:04:14.771 INFO:teuthology.orchestra.run.smithi116.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T01:04:14.845 INFO:teuthology.orchestra.run.smithi008.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2445-gd2c45d97-1jammy [6,775 kB] 2024-04-26T01:04:14.865 INFO:teuthology.orchestra.run.smithi164.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2445-gd2c45d97-1jammy [22.7 MB] 2024-04-26T01:04:15.008 INFO:teuthology.orchestra.run.smithi012.stdout:Get:64 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2445-gd2c45d97-1jammy [244 kB] 2024-04-26T01:04:15.055 INFO:teuthology.orchestra.run.smithi012.stdout:Get:65 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2445-gd2c45d97-1jammy [124 kB] 2024-04-26T01:04:15.055 INFO:teuthology.orchestra.run.smithi012.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T01:04:15.057 INFO:teuthology.orchestra.run.smithi012.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T01:04:15.058 INFO:teuthology.orchestra.run.smithi012.stdout:Get:68 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2445-gd2c45d97-1jammy [1,526 kB] 2024-04-26T01:04:15.060 INFO:teuthology.orchestra.run.smithi012.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T01:04:15.061 INFO:teuthology.orchestra.run.smithi012.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T01:04:15.063 INFO:teuthology.orchestra.run.smithi012.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T01:04:15.167 INFO:teuthology.orchestra.run.smithi012.stdout:Get:72 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2445-gd2c45d97-1jammy [6,245 kB] 2024-04-26T01:04:15.201 INFO:teuthology.orchestra.run.smithi116.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T01:04:15.237 INFO:teuthology.orchestra.run.smithi116.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T01:04:15.245 INFO:teuthology.orchestra.run.smithi116.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T01:04:15.265 INFO:teuthology.orchestra.run.smithi012.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T01:04:15.298 INFO:teuthology.orchestra.run.smithi116.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T01:04:15.323 INFO:teuthology.orchestra.run.smithi116.stdout:Get:34 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2445-gd2c45d97-1jammy [711 kB] 2024-04-26T01:04:15.456 INFO:teuthology.orchestra.run.smithi116.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T01:04:15.480 INFO:teuthology.orchestra.run.smithi116.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T01:04:15.637 INFO:teuthology.orchestra.run.smithi012.stdout:Get:74 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2445-gd2c45d97-1jammy [22.7 MB] 2024-04-26T01:04:15.655 INFO:teuthology.orchestra.run.smithi116.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T01:04:15.657 INFO:teuthology.orchestra.run.smithi116.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2445-gd2c45d97-1jammy [356 kB] 2024-04-26T01:04:15.670 INFO:teuthology.orchestra.run.smithi008.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T01:04:15.695 INFO:teuthology.orchestra.run.smithi012.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T01:04:15.700 INFO:teuthology.orchestra.run.smithi012.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T01:04:15.837 INFO:teuthology.orchestra.run.smithi116.stdout:Get:39 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2445-gd2c45d97-1jammy [32.8 kB] 2024-04-26T01:04:15.846 INFO:teuthology.orchestra.run.smithi116.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2445-gd2c45d97-1jammy [180 kB] 2024-04-26T01:04:15.933 INFO:teuthology.orchestra.run.smithi008.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T01:04:15.940 INFO:teuthology.orchestra.run.smithi116.stdout:Get:41 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2445-gd2c45d97-1jammy [65.8 kB] 2024-04-26T01:04:15.949 INFO:teuthology.orchestra.run.smithi116.stdout:Get:42 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2445-gd2c45d97-1jammy [330 kB] 2024-04-26T01:04:15.987 INFO:teuthology.orchestra.run.smithi116.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T01:04:15.989 INFO:teuthology.orchestra.run.smithi012.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T01:04:15.990 INFO:teuthology.orchestra.run.smithi012.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T01:04:15.992 INFO:teuthology.orchestra.run.smithi012.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T01:04:15.994 INFO:teuthology.orchestra.run.smithi012.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T01:04:15.995 INFO:teuthology.orchestra.run.smithi012.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T01:04:15.997 INFO:teuthology.orchestra.run.smithi012.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T01:04:15.998 INFO:teuthology.orchestra.run.smithi012.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T01:04:16.000 INFO:teuthology.orchestra.run.smithi012.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T01:04:16.027 INFO:teuthology.orchestra.run.smithi116.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T01:04:16.040 INFO:teuthology.orchestra.run.smithi116.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T01:04:16.136 INFO:teuthology.orchestra.run.smithi116.stdout:Get:46 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2445-gd2c45d97-1jammy [6,775 kB] 2024-04-26T01:04:16.181 INFO:teuthology.orchestra.run.smithi012.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T01:04:16.248 INFO:teuthology.orchestra.run.smithi116.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T01:04:16.249 INFO:teuthology.orchestra.run.smithi116.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T01:04:16.255 INFO:teuthology.orchestra.run.smithi116.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T01:04:16.261 INFO:teuthology.orchestra.run.smithi116.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T01:04:16.275 INFO:teuthology.orchestra.run.smithi116.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T01:04:16.296 INFO:teuthology.orchestra.run.smithi116.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T01:04:16.303 INFO:teuthology.orchestra.run.smithi116.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T01:04:16.479 INFO:teuthology.orchestra.run.smithi116.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T01:04:16.595 INFO:teuthology.orchestra.run.smithi012.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T01:04:16.642 INFO:teuthology.orchestra.run.smithi012.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T01:04:16.676 INFO:teuthology.orchestra.run.smithi012.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T01:04:16.743 INFO:teuthology.orchestra.run.smithi116.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T01:04:16.772 INFO:teuthology.orchestra.run.smithi012.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T01:04:16.812 INFO:teuthology.orchestra.run.smithi012.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T01:04:16.816 INFO:teuthology.orchestra.run.smithi164.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2445-gd2c45d97-1jammy [14.0 kB] 2024-04-26T01:04:16.817 INFO:teuthology.orchestra.run.smithi164.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [869 kB] 2024-04-26T01:04:16.824 INFO:teuthology.orchestra.run.smithi012.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T01:04:16.829 INFO:teuthology.orchestra.run.smithi012.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T01:04:16.834 INFO:teuthology.orchestra.run.smithi164.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2445-gd2c45d97-1jammy [2,437 kB] 2024-04-26T01:04:16.876 INFO:teuthology.orchestra.run.smithi012.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T01:04:16.879 INFO:teuthology.orchestra.run.smithi012.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T01:04:16.900 INFO:teuthology.orchestra.run.smithi012.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T01:04:16.921 INFO:teuthology.orchestra.run.smithi116.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T01:04:16.935 INFO:teuthology.orchestra.run.smithi116.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T01:04:16.955 INFO:teuthology.orchestra.run.smithi116.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T01:04:16.989 INFO:teuthology.orchestra.run.smithi164.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2445-gd2c45d97-1jammy [791 kB] 2024-04-26T01:04:17.019 INFO:teuthology.orchestra.run.smithi116.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T01:04:17.056 INFO:teuthology.orchestra.run.smithi164.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2445-gd2c45d97-1jammy [147 kB] 2024-04-26T01:04:17.061 INFO:teuthology.orchestra.run.smithi164.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2445-gd2c45d97-1jammy [2,293 kB] 2024-04-26T01:04:17.064 INFO:teuthology.orchestra.run.smithi012.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T01:04:17.227 INFO:teuthology.orchestra.run.smithi008.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T01:04:17.261 INFO:teuthology.orchestra.run.smithi164.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2445-gd2c45d97-1jammy [8,625 kB] 2024-04-26T01:04:17.310 INFO:teuthology.orchestra.run.smithi116.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T01:04:17.312 INFO:teuthology.orchestra.run.smithi116.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T01:04:17.314 INFO:teuthology.orchestra.run.smithi116.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T01:04:17.315 INFO:teuthology.orchestra.run.smithi116.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T01:04:17.317 INFO:teuthology.orchestra.run.smithi116.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T01:04:17.338 INFO:teuthology.orchestra.run.smithi012.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T01:04:17.357 INFO:teuthology.orchestra.run.smithi012.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T01:04:17.357 INFO:teuthology.orchestra.run.smithi012.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2445-gd2c45d97-1jammy [14.0 kB] 2024-04-26T01:04:17.357 INFO:teuthology.orchestra.run.smithi012.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [869 kB] 2024-04-26T01:04:17.375 INFO:teuthology.orchestra.run.smithi012.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2445-gd2c45d97-1jammy [2,437 kB] 2024-04-26T01:04:17.407 INFO:teuthology.orchestra.run.smithi008.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T01:04:17.421 INFO:teuthology.orchestra.run.smithi008.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T01:04:17.431 INFO:teuthology.orchestra.run.smithi008.stdout:Get:19 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2445-gd2c45d97-1jammy [112 kB] 2024-04-26T01:04:17.441 INFO:teuthology.orchestra.run.smithi008.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T01:04:17.498 INFO:teuthology.orchestra.run.smithi012.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2445-gd2c45d97-1jammy [791 kB] 2024-04-26T01:04:17.508 INFO:teuthology.orchestra.run.smithi008.stdout:Get:21 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2445-gd2c45d97-1jammy [467 kB] 2024-04-26T01:04:17.509 INFO:teuthology.orchestra.run.smithi012.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T01:04:17.536 INFO:teuthology.orchestra.run.smithi116.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T01:04:17.538 INFO:teuthology.orchestra.run.smithi012.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T01:04:17.562 INFO:teuthology.orchestra.run.smithi012.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2445-gd2c45d97-1jammy [147 kB] 2024-04-26T01:04:17.567 INFO:teuthology.orchestra.run.smithi012.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2445-gd2c45d97-1jammy [2,293 kB] 2024-04-26T01:04:17.695 INFO:teuthology.orchestra.run.smithi012.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T01:04:17.698 INFO:teuthology.orchestra.run.smithi012.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T01:04:17.700 INFO:teuthology.orchestra.run.smithi012.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T01:04:17.719 INFO:teuthology.orchestra.run.smithi012.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2445-gd2c45d97-1jammy [8,625 kB] 2024-04-26T01:04:17.722 INFO:teuthology.orchestra.run.smithi012.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T01:04:17.727 INFO:teuthology.orchestra.run.smithi012.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T01:04:17.864 INFO:teuthology.orchestra.run.smithi008.stdout:Get:22 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2445-gd2c45d97-1jammy [25.6 MB] 2024-04-26T01:04:17.934 INFO:teuthology.orchestra.run.smithi164.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2445-gd2c45d97-1jammy [14.2 kB] 2024-04-26T01:04:17.935 INFO:teuthology.orchestra.run.smithi164.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2445-gd2c45d97-1jammy [50.7 MB] 2024-04-26T01:04:17.985 INFO:teuthology.orchestra.run.smithi116.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T01:04:18.004 INFO:teuthology.orchestra.run.smithi116.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T01:04:18.019 INFO:teuthology.orchestra.run.smithi008.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T01:04:18.318 INFO:teuthology.orchestra.run.smithi012.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2445-gd2c45d97-1jammy [14.2 kB] 2024-04-26T01:04:18.319 INFO:teuthology.orchestra.run.smithi012.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2445-gd2c45d97-1jammy [50.7 MB] 2024-04-26T01:04:18.389 INFO:teuthology.orchestra.run.smithi008.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T01:04:18.855 INFO:teuthology.orchestra.run.smithi008.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T01:04:18.928 INFO:teuthology.orchestra.run.smithi008.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T01:04:18.932 INFO:teuthology.orchestra.run.smithi008.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T01:04:19.225 INFO:teuthology.orchestra.run.smithi008.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T01:04:19.585 INFO:teuthology.orchestra.run.smithi008.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T01:04:19.883 INFO:teuthology.orchestra.run.smithi008.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T01:04:19.953 INFO:teuthology.orchestra.run.smithi008.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T01:04:19.957 INFO:teuthology.orchestra.run.smithi008.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T01:04:20.102 INFO:teuthology.orchestra.run.smithi008.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T01:04:20.107 INFO:teuthology.orchestra.run.smithi008.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T01:04:20.218 INFO:teuthology.orchestra.run.smithi116.stdout:Get:68 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2445-gd2c45d97-1jammy [112 kB] 2024-04-26T01:04:20.269 INFO:teuthology.orchestra.run.smithi116.stdout:Get:69 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2445-gd2c45d97-1jammy [467 kB] 2024-04-26T01:04:20.289 INFO:teuthology.orchestra.run.smithi008.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T01:04:20.372 INFO:teuthology.orchestra.run.smithi008.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T01:04:20.401 INFO:teuthology.orchestra.run.smithi008.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T01:04:20.418 INFO:teuthology.orchestra.run.smithi008.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T01:04:20.463 INFO:teuthology.orchestra.run.smithi008.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T01:04:20.501 INFO:teuthology.orchestra.run.smithi116.stdout:Get:70 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2445-gd2c45d97-1jammy [25.6 MB] 2024-04-26T01:04:20.678 INFO:teuthology.orchestra.run.smithi008.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T01:04:20.693 INFO:teuthology.orchestra.run.smithi008.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T01:04:20.700 INFO:teuthology.orchestra.run.smithi008.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T01:04:20.706 INFO:teuthology.orchestra.run.smithi008.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T01:04:20.727 INFO:teuthology.orchestra.run.smithi008.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T01:04:20.731 INFO:teuthology.orchestra.run.smithi008.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T01:04:20.911 INFO:teuthology.orchestra.run.smithi008.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T01:04:21.258 INFO:teuthology.orchestra.run.smithi008.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T01:04:21.303 INFO:teuthology.orchestra.run.smithi008.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T01:04:21.328 INFO:teuthology.orchestra.run.smithi008.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T01:04:22.144 INFO:teuthology.orchestra.run.smithi012.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2445-gd2c45d97-1jammy [121 kB] 2024-04-26T01:04:22.146 INFO:teuthology.orchestra.run.smithi012.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2445-gd2c45d97-1jammy [38.9 kB] 2024-04-26T01:04:22.147 INFO:teuthology.orchestra.run.smithi012.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2445-gd2c45d97-1jammy [13.4 MB] 2024-04-26T01:04:22.168 INFO:teuthology.orchestra.run.smithi008.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T01:04:22.169 INFO:teuthology.orchestra.run.smithi164.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2445-gd2c45d97-1jammy [121 kB] 2024-04-26T01:04:22.172 INFO:teuthology.orchestra.run.smithi164.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2445-gd2c45d97-1jammy [38.9 kB] 2024-04-26T01:04:22.172 INFO:teuthology.orchestra.run.smithi008.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T01:04:22.173 INFO:teuthology.orchestra.run.smithi164.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2445-gd2c45d97-1jammy [13.4 MB] 2024-04-26T01:04:22.192 INFO:teuthology.orchestra.run.smithi008.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T01:04:22.302 INFO:teuthology.orchestra.run.smithi008.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T01:04:22.545 INFO:teuthology.orchestra.run.smithi008.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T01:04:22.698 INFO:teuthology.orchestra.run.smithi008.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T01:04:22.734 INFO:teuthology.orchestra.run.smithi116.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T01:04:22.777 INFO:teuthology.orchestra.run.smithi116.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T01:04:22.863 INFO:teuthology.orchestra.run.smithi008.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T01:04:22.890 INFO:teuthology.orchestra.run.smithi012.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [91.4 kB] 2024-04-26T01:04:22.944 INFO:teuthology.orchestra.run.smithi116.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T01:04:23.005 INFO:teuthology.orchestra.run.smithi164.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [91.4 kB] 2024-04-26T01:04:23.058 INFO:teuthology.orchestra.run.smithi116.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T01:04:23.160 INFO:teuthology.orchestra.run.smithi116.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T01:04:23.216 INFO:teuthology.orchestra.run.smithi008.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T01:04:23.300 INFO:teuthology.orchestra.run.smithi116.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T01:04:23.329 INFO:teuthology.orchestra.run.smithi116.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T01:04:23.370 INFO:teuthology.orchestra.run.smithi116.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T01:04:23.477 INFO:teuthology.orchestra.run.smithi012.stdout:Fetched 179 MB in 12s (14.9 MB/s) 2024-04-26T01:04:23.483 INFO:teuthology.orchestra.run.smithi008.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T01:04:23.572 INFO:teuthology.orchestra.run.smithi164.stdout:Fetched 179 MB in 12s (14.7 MB/s) 2024-04-26T01:04:23.602 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T01:04:23.664 INFO:teuthology.orchestra.run.smithi008.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T01:04:23.678 INFO:teuthology.orchestra.run.smithi008.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T01:04:23.698 INFO:teuthology.orchestra.run.smithi008.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T01:04:23.703 INFO:teuthology.orchestra.run.smithi012.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T01:04:23.709 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T01:04:23.730 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T01:04:23.763 INFO:teuthology.orchestra.run.smithi008.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T01:04:23.784 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:04:23.825 INFO:teuthology.orchestra.run.smithi164.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T01:04:23.830 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T01:04:23.912 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:04:24.055 INFO:teuthology.orchestra.run.smithi008.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T01:04:24.057 INFO:teuthology.orchestra.run.smithi008.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T01:04:24.062 INFO:teuthology.orchestra.run.smithi008.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T01:04:24.062 INFO:teuthology.orchestra.run.smithi008.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T01:04:24.063 INFO:teuthology.orchestra.run.smithi008.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T01:04:24.071 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T01:04:24.099 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T01:04:24.138 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:04:24.215 INFO:teuthology.orchestra.run.smithi116.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T01:04:24.240 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T01:04:24.267 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T01:04:24.284 INFO:teuthology.orchestra.run.smithi008.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T01:04:24.307 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:04:24.499 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T01:04:24.526 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T01:04:24.566 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:04:24.660 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T01:04:24.687 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T01:04:24.726 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:04:24.808 INFO:teuthology.orchestra.run.smithi116.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T01:04:24.884 INFO:teuthology.orchestra.run.smithi008.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T01:04:24.902 INFO:teuthology.orchestra.run.smithi008.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T01:04:25.003 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T01:04:25.030 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:04:25.078 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:04:25.112 INFO:teuthology.orchestra.run.smithi116.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T01:04:25.163 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T01:04:25.190 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:04:25.228 INFO:teuthology.orchestra.run.smithi116.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T01:04:25.238 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:04:25.498 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T01:04:25.524 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:04:25.564 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:04:25.658 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T01:04:25.684 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:04:25.725 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:04:26.018 INFO:teuthology.orchestra.run.smithi116.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T01:04:26.034 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T01:04:26.061 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:04:26.101 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:04:26.103 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T01:04:26.129 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:04:26.169 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:04:26.429 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T01:04:26.453 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T01:04:26.456 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T01:04:26.482 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T01:04:26.487 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:04:26.514 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:04:26.579 INFO:teuthology.orchestra.run.smithi116.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T01:04:26.775 INFO:teuthology.orchestra.run.smithi116.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T01:04:26.822 INFO:teuthology.orchestra.run.smithi116.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T01:04:26.840 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package librados2. 2024-04-26T01:04:26.859 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package librados2. 2024-04-26T01:04:26.867 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../007-librados2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:26.885 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../007-librados2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:26.907 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:26.917 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:27.368 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package librbd1. 2024-04-26T01:04:27.395 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../008-librbd1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:27.435 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:27.545 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package librbd1. 2024-04-26T01:04:27.571 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../008-librbd1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:27.573 INFO:teuthology.orchestra.run.smithi008.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2445-gd2c45d97-1jammy [5,084 kB] 2024-04-26T01:04:27.612 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:27.871 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T01:04:27.887 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:27.921 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:28.081 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T01:04:28.107 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:28.145 INFO:teuthology.orchestra.run.smithi116.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T01:04:28.149 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:28.182 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rados. 2024-04-26T01:04:28.209 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:28.248 INFO:teuthology.orchestra.run.smithi116.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T01:04:28.249 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:28.434 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rados. 2024-04-26T01:04:28.449 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:28.484 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:28.752 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T01:04:28.753 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T01:04:28.768 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:04:28.778 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:04:28.811 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:28.812 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:28.883 INFO:teuthology.orchestra.run.smithi008.stdout:Get:72 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2445-gd2c45d97-1jammy [244 kB] 2024-04-26T01:04:28.884 INFO:teuthology.orchestra.run.smithi116.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2445-gd2c45d97-1jammy [5,084 kB] 2024-04-26T01:04:28.953 INFO:teuthology.orchestra.run.smithi008.stdout:Get:73 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2445-gd2c45d97-1jammy [124 kB] 2024-04-26T01:04:28.969 INFO:teuthology.orchestra.run.smithi008.stdout:Get:74 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2445-gd2c45d97-1jammy [1,526 kB] 2024-04-26T01:04:29.031 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T01:04:29.047 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:29.064 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T01:04:29.079 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:29.089 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:29.122 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:29.140 INFO:teuthology.orchestra.run.smithi008.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T01:04:29.151 INFO:teuthology.orchestra.run.smithi008.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T01:04:29.169 INFO:teuthology.orchestra.run.smithi116.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T01:04:29.287 INFO:teuthology.orchestra.run.smithi008.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2445-gd2c45d97-1jammy [6,245 kB] 2024-04-26T01:04:29.314 INFO:teuthology.orchestra.run.smithi008.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T01:04:29.345 INFO:teuthology.orchestra.run.smithi116.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T01:04:29.404 INFO:teuthology.orchestra.run.smithi008.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T01:04:29.408 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T01:04:29.436 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:04:29.459 INFO:teuthology.orchestra.run.smithi008.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T01:04:29.467 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T01:04:29.475 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:29.492 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:04:29.533 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:29.598 INFO:teuthology.orchestra.run.smithi008.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T01:04:29.619 INFO:teuthology.orchestra.run.smithi116.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T01:04:29.637 INFO:teuthology.orchestra.run.smithi116.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T01:04:29.656 INFO:teuthology.orchestra.run.smithi008.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T01:04:29.696 INFO:teuthology.orchestra.run.smithi008.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T01:04:29.795 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T01:04:29.822 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T01:04:29.853 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T01:04:29.870 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:04:29.879 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T01:04:29.920 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:04:29.922 INFO:teuthology.orchestra.run.smithi116.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T01:04:30.015 INFO:teuthology.orchestra.run.smithi116.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2445-gd2c45d97-1jammy [244 kB] 2024-04-26T01:04:30.092 INFO:teuthology.orchestra.run.smithi116.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2445-gd2c45d97-1jammy [124 kB] 2024-04-26T01:04:30.102 INFO:teuthology.orchestra.run.smithi116.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2445-gd2c45d97-1jammy [1,526 kB] 2024-04-26T01:04:30.122 INFO:teuthology.orchestra.run.smithi116.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T01:04:30.164 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T01:04:30.191 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T01:04:30.197 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T01:04:30.212 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T01:04:30.231 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T01:04:30.247 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T01:04:30.437 INFO:teuthology.orchestra.run.smithi116.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2445-gd2c45d97-1jammy [6,245 kB] 2024-04-26T01:04:30.475 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T01:04:30.500 INFO:teuthology.orchestra.run.smithi008.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2445-gd2c45d97-1jammy [22.7 MB] 2024-04-26T01:04:30.500 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T01:04:30.502 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:30.515 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:30.542 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:30.558 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:30.861 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T01:04:30.864 INFO:teuthology.orchestra.run.smithi008.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T01:04:30.887 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T01:04:30.888 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T01:04:30.913 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T01:04:30.936 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:04:30.961 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:04:31.197 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T01:04:31.224 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T01:04:31.247 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T01:04:31.263 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:04:31.264 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T01:04:31.285 INFO:teuthology.orchestra.run.smithi116.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2445-gd2c45d97-1jammy [22.7 MB] 2024-04-26T01:04:31.292 INFO:teuthology.orchestra.run.smithi008.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T01:04:31.305 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:04:31.473 INFO:teuthology.orchestra.run.smithi008.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T01:04:31.566 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T01:04:31.582 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T01:04:31.600 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T01:04:31.606 INFO:teuthology.orchestra.run.smithi008.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T01:04:31.625 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:04:31.627 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T01:04:31.666 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:04:31.724 INFO:teuthology.orchestra.run.smithi116.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T01:04:31.839 INFO:teuthology.orchestra.run.smithi116.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T01:04:31.911 INFO:teuthology.orchestra.run.smithi116.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T01:04:31.953 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package lua5.1. 2024-04-26T01:04:31.978 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package lua5.1. 2024-04-26T01:04:31.980 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T01:04:32.005 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T01:04:32.020 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:04:32.045 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:04:32.289 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package lua-any. 2024-04-26T01:04:32.305 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T01:04:32.348 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T01:04:32.356 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package lua-any. 2024-04-26T01:04:32.370 INFO:teuthology.orchestra.run.smithi116.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T01:04:32.383 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T01:04:32.423 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T01:04:32.429 INFO:teuthology.orchestra.run.smithi008.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T01:04:32.501 INFO:teuthology.orchestra.run.smithi116.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T01:04:32.658 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package zip. 2024-04-26T01:04:32.674 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T01:04:32.708 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T01:04:32.725 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package zip. 2024-04-26T01:04:32.752 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T01:04:32.792 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T01:04:32.886 INFO:teuthology.orchestra.run.smithi008.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2445-gd2c45d97-1jammy [14.0 kB] 2024-04-26T01:04:32.887 INFO:teuthology.orchestra.run.smithi008.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [869 kB] 2024-04-26T01:04:32.959 INFO:teuthology.orchestra.run.smithi008.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2445-gd2c45d97-1jammy [2,437 kB] 2024-04-26T01:04:32.978 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package unzip. 2024-04-26T01:04:32.980 INFO:teuthology.orchestra.run.smithi116.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2445-gd2c45d97-1jammy [14.0 kB] 2024-04-26T01:04:32.981 INFO:teuthology.orchestra.run.smithi116.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [869 kB] 2024-04-26T01:04:33.005 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T01:04:33.045 INFO:teuthology.orchestra.run.smithi116.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2445-gd2c45d97-1jammy [2,437 kB] 2024-04-26T01:04:33.053 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:04:33.130 INFO:teuthology.orchestra.run.smithi008.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T01:04:33.145 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package unzip. 2024-04-26T01:04:33.172 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T01:04:33.212 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:04:33.262 INFO:teuthology.orchestra.run.smithi116.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2445-gd2c45d97-1jammy [791 kB] 2024-04-26T01:04:33.320 INFO:teuthology.orchestra.run.smithi116.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2445-gd2c45d97-1jammy [147 kB] 2024-04-26T01:04:33.325 INFO:teuthology.orchestra.run.smithi116.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2445-gd2c45d97-1jammy [2,293 kB] 2024-04-26T01:04:33.404 INFO:teuthology.orchestra.run.smithi008.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T01:04:33.409 INFO:teuthology.orchestra.run.smithi008.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2445-gd2c45d97-1jammy [791 kB] 2024-04-26T01:04:33.457 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package luarocks. 2024-04-26T01:04:33.484 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T01:04:33.493 INFO:teuthology.orchestra.run.smithi008.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T01:04:33.507 INFO:teuthology.orchestra.run.smithi116.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2445-gd2c45d97-1jammy [8,625 kB] 2024-04-26T01:04:33.515 INFO:teuthology.orchestra.run.smithi008.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2445-gd2c45d97-1jammy [147 kB] 2024-04-26T01:04:33.530 INFO:teuthology.orchestra.run.smithi008.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2445-gd2c45d97-1jammy [2,293 kB] 2024-04-26T01:04:33.557 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package luarocks. 2024-04-26T01:04:33.584 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T01:04:33.615 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:04:33.623 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:04:33.875 INFO:teuthology.orchestra.run.smithi008.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2445-gd2c45d97-1jammy [8,625 kB] 2024-04-26T01:04:33.975 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package librgw2. 2024-04-26T01:04:33.998 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package librgw2. 2024-04-26T01:04:34.002 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../025-librgw2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:34.026 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../025-librgw2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:34.042 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:34.064 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:34.072 INFO:teuthology.orchestra.run.smithi116.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2445-gd2c45d97-1jammy [14.2 kB] 2024-04-26T01:04:34.073 INFO:teuthology.orchestra.run.smithi116.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2445-gd2c45d97-1jammy [50.7 MB] 2024-04-26T01:04:34.298 INFO:teuthology.orchestra.run.smithi008.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T01:04:34.431 INFO:teuthology.orchestra.run.smithi008.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T01:04:34.594 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T01:04:34.609 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:34.650 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T01:04:34.661 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:34.678 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:34.717 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:34.947 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T01:04:34.969 INFO:teuthology.orchestra.run.smithi008.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2445-gd2c45d97-1jammy [14.2 kB] 2024-04-26T01:04:34.970 INFO:teuthology.orchestra.run.smithi008.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2445-gd2c45d97-1jammy [50.7 MB] 2024-04-26T01:04:34.974 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T01:04:34.978 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T01:04:34.994 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T01:04:35.005 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:04:35.028 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:04:35.057 INFO:teuthology.orchestra.run.smithi008.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T01:04:35.225 INFO:teuthology.orchestra.run.smithi008.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T01:04:35.300 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T01:04:35.327 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:35.348 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T01:04:35.367 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:35.375 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:35.414 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:35.498 INFO:teuthology.orchestra.run.smithi008.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T01:04:35.516 INFO:teuthology.orchestra.run.smithi008.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T01:04:35.670 INFO:teuthology.orchestra.run.smithi008.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T01:04:35.699 INFO:teuthology.orchestra.run.smithi008.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T01:04:35.803 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-common. 2024-04-26T01:04:35.829 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:35.842 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-common. 2024-04-26T01:04:35.857 INFO:teuthology.orchestra.run.smithi008.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T01:04:35.859 INFO:teuthology.orchestra.run.smithi008.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T01:04:35.861 INFO:teuthology.orchestra.run.smithi008.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T01:04:35.869 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:35.870 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:35.883 INFO:teuthology.orchestra.run.smithi008.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T01:04:35.889 INFO:teuthology.orchestra.run.smithi008.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T01:04:35.909 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:37.049 INFO:teuthology.orchestra.run.smithi116.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2445-gd2c45d97-1jammy [121 kB] 2024-04-26T01:04:37.052 INFO:teuthology.orchestra.run.smithi116.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2445-gd2c45d97-1jammy [38.9 kB] 2024-04-26T01:04:37.053 INFO:teuthology.orchestra.run.smithi116.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2445-gd2c45d97-1jammy [13.4 MB] 2024-04-26T01:04:37.158 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-base. 2024-04-26T01:04:37.184 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:37.206 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-base. 2024-04-26T01:04:37.225 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:37.233 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:37.274 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:37.548 INFO:teuthology.orchestra.run.smithi116.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [91.4 kB] 2024-04-26T01:04:37.843 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T01:04:37.871 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T01:04:37.910 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:04:37.954 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T01:04:37.969 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T01:04:38.012 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:04:38.106 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 179 MB in 26s (6,778 kB/s) 2024-04-26T01:04:38.238 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T01:04:38.262 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T01:04:38.265 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T01:04:38.304 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:04:38.315 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T01:04:38.341 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T01:04:38.361 INFO:teuthology.orchestra.run.smithi116.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T01:04:38.370 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T01:04:38.381 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:04:38.460 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:04:38.709 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T01:04:38.736 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T01:04:38.776 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:04:38.805 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T01:04:38.832 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T01:04:38.872 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:04:38.891 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T01:04:38.918 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T01:04:38.957 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:04:39.087 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T01:04:39.114 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T01:04:39.153 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:04:39.158 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T01:04:39.185 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T01:04:39.216 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:04:39.235 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T01:04:39.263 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T01:04:39.293 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:04:39.432 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T01:04:39.458 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T01:04:39.489 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:04:39.580 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T01:04:39.607 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T01:04:39.646 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:04:39.678 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T01:04:39.704 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:04:39.753 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:04:39.759 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T01:04:39.785 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T01:04:39.825 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T01:04:39.932 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T01:04:39.960 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T01:04:40.001 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T01:04:40.153 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-portend. 2024-04-26T01:04:40.164 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T01:04:40.180 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T01:04:40.191 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:04:40.220 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T01:04:40.231 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:04:40.302 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-portend. 2024-04-26T01:04:40.329 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T01:04:40.368 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T01:04:40.522 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T01:04:40.550 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T01:04:40.559 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T01:04:40.585 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:04:40.589 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T01:04:40.625 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:04:40.688 INFO:teuthology.orchestra.run.smithi008.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2445-gd2c45d97-1jammy [121 kB] 2024-04-26T01:04:40.691 INFO:teuthology.orchestra.run.smithi008.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2445-gd2c45d97-1jammy [38.9 kB] 2024-04-26T01:04:40.692 INFO:teuthology.orchestra.run.smithi008.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2445-gd2c45d97-1jammy [13.4 MB] 2024-04-26T01:04:40.697 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T01:04:40.723 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T01:04:40.771 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T01:04:40.850 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T01:04:40.874 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T01:04:40.917 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:04:40.954 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T01:04:40.980 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T01:04:41.012 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:04:41.091 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T01:04:41.117 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T01:04:41.174 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:04:41.315 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librados2. 2024-04-26T01:04:41.321 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T01:04:41.341 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../007-librados2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:41.347 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T01:04:41.381 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:41.395 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T01:04:41.519 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T01:04:41.546 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T01:04:41.594 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T01:04:41.754 INFO:teuthology.orchestra.run.smithi008.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [91.4 kB] 2024-04-26T01:04:41.782 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T01:04:41.809 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T01:04:41.834 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librbd1. 2024-04-26T01:04:41.857 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T01:04:41.860 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../008-librbd1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:41.900 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:41.989 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T01:04:42.016 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T01:04:42.064 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T01:04:42.159 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T01:04:42.186 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T01:04:42.226 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:04:42.337 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T01:04:42.342 INFO:teuthology.orchestra.run.smithi008.stdout:Fetched 179 MB in 31s (5,805 kB/s) 2024-04-26T01:04:42.358 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T01:04:42.363 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:42.385 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T01:04:42.403 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:42.425 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:04:42.510 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T01:04:42.554 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-mako. 2024-04-26T01:04:42.581 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T01:04:42.614 INFO:teuthology.orchestra.run.smithi008.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T01:04:42.619 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T01:04:42.629 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:04:42.658 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rados. 2024-04-26T01:04:42.671 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:42.706 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:42.728 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-mako. 2024-04-26T01:04:42.755 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T01:04:42.794 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:04:42.819 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:04:43.017 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T01:04:43.043 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:04:43.098 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T01:04:43.125 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T01:04:43.164 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:04:43.179 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:43.509 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T01:04:43.536 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T01:04:43.576 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:04:43.774 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T01:04:43.801 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T01:04:43.806 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T01:04:43.834 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T01:04:43.873 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:04:44.007 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:04:44.070 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T01:04:44.085 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:44.128 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:44.193 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T01:04:44.219 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T01:04:44.259 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:04:44.277 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T01:04:44.303 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T01:04:44.335 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:04:44.447 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T01:04:44.473 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:04:44.514 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:44.553 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-webob. 2024-04-26T01:04:44.581 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T01:04:44.620 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:04:44.630 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-webob. 2024-04-26T01:04:44.656 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T01:04:44.688 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T01:04:44.698 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:04:44.716 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:04:44.772 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:04:44.850 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T01:04:44.876 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T01:04:44.917 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:04:44.965 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T01:04:44.992 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T01:04:45.032 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T01:04:45.034 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:04:45.059 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T01:04:45.102 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:04:45.158 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T01:04:45.185 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:04:45.194 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T01:04:45.210 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T01:04:45.225 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:04:45.244 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T01:04:45.326 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T01:04:45.353 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T01:04:45.392 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:04:45.393 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T01:04:45.409 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T01:04:45.443 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:04:45.489 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T01:04:45.515 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:45.555 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:45.603 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T01:04:45.630 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:04:45.670 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:04:45.695 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-paste. 2024-04-26T01:04:45.722 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T01:04:45.746 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-paste. 2024-04-26T01:04:45.773 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T01:04:45.782 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:04:45.813 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:04:45.900 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T01:04:45.926 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T01:04:45.966 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:04:46.032 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T01:04:46.059 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T01:04:46.098 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:04:46.098 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T01:04:46.107 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T01:04:46.114 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T01:04:46.123 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T01:04:46.156 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:04:46.157 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:04:46.277 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T01:04:46.292 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T01:04:46.335 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:04:46.384 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package librados2. 2024-04-26T01:04:46.411 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../007-librados2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:46.443 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:46.452 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T01:04:46.459 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T01:04:46.478 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T01:04:46.486 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T01:04:46.518 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:04:46.525 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:04:46.597 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T01:04:46.612 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T01:04:46.647 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:04:46.812 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T01:04:46.821 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T01:04:46.839 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T01:04:46.848 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T01:04:46.870 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T01:04:46.888 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package librbd1. 2024-04-26T01:04:46.888 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T01:04:46.914 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../008-librbd1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:46.954 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:47.000 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua5.1. 2024-04-26T01:04:47.026 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T01:04:47.066 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:04:47.157 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T01:04:47.184 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T01:04:47.199 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T01:04:47.223 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:04:47.226 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T01:04:47.266 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:04:47.386 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua-any. 2024-04-26T01:04:47.401 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T01:04:47.444 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T01:04:47.549 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T01:04:47.561 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T01:04:47.575 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:47.588 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T01:04:47.615 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:47.628 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T01:04:47.662 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T01:04:47.663 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package zip. 2024-04-26T01:04:47.679 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T01:04:47.689 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T01:04:47.713 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T01:04:47.720 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T01:04:47.876 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-rados. 2024-04-26T01:04:47.892 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:47.926 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:47.956 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-idna. 2024-04-26T01:04:47.983 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T01:04:48.022 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T01:04:48.023 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-idna. 2024-04-26T01:04:48.050 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T01:04:48.090 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T01:04:48.208 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package unzip. 2024-04-26T01:04:48.234 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T01:04:48.262 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T01:04:48.275 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:04:48.289 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:04:48.329 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:48.333 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T01:04:48.361 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T01:04:48.400 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:04:48.417 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T01:04:48.444 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T01:04:48.484 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:04:48.582 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T01:04:48.598 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:48.640 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:48.670 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package luarocks. 2024-04-26T01:04:48.697 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T01:04:48.737 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-requests. 2024-04-26T01:04:48.737 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:04:48.764 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T01:04:48.803 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:04:48.812 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-requests. 2024-04-26T01:04:48.839 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T01:04:48.879 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:04:48.959 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T01:04:48.987 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:04:49.026 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:49.072 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T01:04:49.097 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T01:04:49.128 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librgw2. 2024-04-26T01:04:49.155 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../025-librgw2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:49.190 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T01:04:49.194 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:49.216 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T01:04:49.256 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:04:49.256 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:04:49.371 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T01:04:49.398 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T01:04:49.429 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:04:49.526 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T01:04:49.542 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:04:49.551 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T01:04:49.576 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:49.578 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:04:49.634 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:49.705 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T01:04:49.720 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:49.724 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T01:04:49.751 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T01:04:49.755 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:49.790 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T01:04:49.956 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T01:04:49.983 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:50.014 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T01:04:50.022 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:50.031 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:50.051 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T01:04:50.067 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:50.072 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:50.083 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T01:04:50.109 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T01:04:50.110 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:50.149 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:04:50.342 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T01:04:50.368 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T01:04:50.391 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T01:04:50.408 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:04:50.419 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T01:04:50.458 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:04:50.472 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T01:04:50.477 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T01:04:50.499 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T01:04:50.504 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:50.538 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:04:50.544 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:50.711 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T01:04:50.738 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T01:04:50.761 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T01:04:50.777 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T01:04:50.788 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T01:04:50.799 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T01:04:50.815 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T01:04:50.827 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T01:04:50.857 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:04:50.972 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-common. 2024-04-26T01:04:50.998 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:51.039 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:51.064 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T01:04:51.079 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:51.130 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T01:04:51.131 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:51.158 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:51.168 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T01:04:51.184 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T01:04:51.197 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:51.227 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:04:51.500 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T01:04:51.527 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:51.567 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:51.572 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package lua5.1. 2024-04-26T01:04:51.575 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T01:04:51.599 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T01:04:51.602 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:51.639 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:04:51.641 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:51.909 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package lua-any. 2024-04-26T01:04:51.925 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T01:04:51.967 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T01:04:52.179 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T01:04:52.206 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:52.211 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package zip. 2024-04-26T01:04:52.227 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T01:04:52.245 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:52.269 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T01:04:52.312 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T01:04:52.339 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:52.387 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:52.435 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-base. 2024-04-26T01:04:52.462 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:52.511 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:52.714 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package unzip. 2024-04-26T01:04:52.742 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T01:04:52.781 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:04:53.065 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T01:04:53.091 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T01:04:53.131 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:04:53.143 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package luarocks. 2024-04-26T01:04:53.170 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T01:04:53.202 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:04:53.373 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph. 2024-04-26T01:04:53.389 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../066-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:53.431 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:53.434 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T01:04:53.460 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T01:04:53.500 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:04:53.523 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph. 2024-04-26T01:04:53.539 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../066-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:53.581 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:53.585 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package librgw2. 2024-04-26T01:04:53.612 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../025-librgw2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:53.651 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:53.829 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T01:04:53.855 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T01:04:53.876 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T01:04:53.903 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:53.903 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:04:53.917 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T01:04:53.944 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:53.950 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:54.017 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:54.173 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T01:04:54.199 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T01:04:54.231 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:04:54.237 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T01:04:54.253 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:54.296 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:54.321 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T01:04:54.348 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:54.387 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:54.388 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T01:04:54.414 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:54.462 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:54.500 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T01:04:54.527 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T01:04:54.567 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:04:54.582 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T01:04:54.609 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T01:04:54.649 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:04:54.815 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package cephadm. 2024-04-26T01:04:54.842 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../069-cephadm_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:54.881 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:54.889 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T01:04:54.915 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T01:04:54.923 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package cephadm. 2024-04-26T01:04:54.935 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T01:04:54.950 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../069-cephadm_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:54.955 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T01:04:54.962 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:54.998 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:55.001 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:55.176 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T01:04:55.203 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T01:04:55.216 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-portend. 2024-04-26T01:04:55.231 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T01:04:55.242 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T01:04:55.266 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T01:04:55.309 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T01:04:55.336 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T01:04:55.376 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T01:04:55.430 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-common. 2024-04-26T01:04:55.457 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:55.496 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:55.544 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T01:04:55.571 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T01:04:55.587 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T01:04:55.610 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T01:04:55.614 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:04:55.653 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:04:55.729 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T01:04:55.756 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:04:55.795 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:04:55.913 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T01:04:55.940 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T01:04:55.980 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:04:56.308 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T01:04:56.335 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T01:04:56.375 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T01:04:56.650 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-babel. 2024-04-26T01:04:56.677 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:04:56.708 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:04:56.711 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T01:04:56.738 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T01:04:56.778 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T01:04:56.783 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-babel. 2024-04-26T01:04:56.811 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:04:56.827 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-base. 2024-04-26T01:04:56.852 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:04:56.854 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:04:56.903 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:56.994 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T01:04:57.021 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T01:04:57.052 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:04:57.081 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T01:04:57.107 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T01:04:57.147 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:04:57.161 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T01:04:57.190 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T01:04:57.219 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:04:57.364 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T01:04:57.391 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:04:57.408 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-mako. 2024-04-26T01:04:57.430 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:57.435 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T01:04:57.465 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T01:04:57.475 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:04:57.492 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T01:04:57.556 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T01:04:57.572 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:04:57.574 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:04:57.605 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:57.750 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T01:04:57.753 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T01:04:57.777 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T01:04:57.779 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T01:04:57.811 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:04:57.816 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T01:04:57.925 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T01:04:57.941 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T01:04:57.975 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T01:04:57.993 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T01:04:58.020 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T01:04:58.059 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:04:58.098 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T01:04:58.119 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-routes. 2024-04-26T01:04:58.124 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T01:04:58.146 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T01:04:58.164 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:04:58.185 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:04:58.253 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-routes. 2024-04-26T01:04:58.281 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T01:04:58.320 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:04:58.371 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T01:04:58.398 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T01:04:58.438 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:04:58.442 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-webob. 2024-04-26T01:04:58.468 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T01:04:58.508 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:04:58.539 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T01:04:58.566 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:04:58.605 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:58.614 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T01:04:58.631 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:04:58.664 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:04:58.715 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T01:04:58.742 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T01:04:58.782 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:04:58.861 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T01:04:58.887 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T01:04:58.937 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:04:59.144 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T01:04:59.171 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T01:04:59.210 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:04:59.248 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T01:04:59.274 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T01:04:59.314 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:04:59.522 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T01:04:59.549 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T01:04:59.588 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T01:04:59.592 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-paste. 2024-04-26T01:04:59.618 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T01:04:59.658 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:04:59.844 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T01:04:59.859 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T01:04:59.866 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-portend. 2024-04-26T01:04:59.894 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T01:04:59.902 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:04:59.933 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T01:04:59.980 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T01:05:00.004 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T01:05:00.047 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:05:00.194 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T01:05:00.195 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T01:05:00.221 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T01:05:00.223 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T01:05:00.261 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T01:05:00.262 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:05:00.314 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T01:05:00.341 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T01:05:00.372 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:05:00.389 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T01:05:00.417 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T01:05:00.456 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:05:00.581 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T01:05:00.608 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T01:05:00.651 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T01:05:00.658 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:05:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T01:05:00.717 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T01:05:00.765 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T01:05:00.793 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T01:05:00.817 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T01:05:00.832 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:05:00.845 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T01:05:00.884 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:05:00.934 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T01:05:00.950 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T01:05:00.984 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T01:05:01.062 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T01:05:01.088 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T01:05:01.128 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:05:01.186 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T01:05:01.214 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T01:05:01.218 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T01:05:01.246 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T01:05:01.261 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:05:01.279 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T01:05:01.285 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:05:01.306 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T01:05:01.354 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T01:05:01.491 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T01:05:01.517 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T01:05:01.583 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T01:05:01.588 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T01:05:01.615 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T01:05:01.654 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:05:01.657 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T01:05:01.684 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T01:05:01.724 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:05:01.993 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-mako. 2024-04-26T01:05:01.995 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T01:05:02.009 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T01:05:02.022 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:02.028 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-idna. 2024-04-26T01:05:02.043 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:05:02.054 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T01:05:02.095 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T01:05:02.155 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:02.363 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T01:05:02.380 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:02.380 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T01:05:02.381 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T01:05:02.396 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T01:05:02.407 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T01:05:02.430 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:05:02.455 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:05:02.502 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:02.725 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-requests. 2024-04-26T01:05:02.752 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T01:05:02.758 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T01:05:02.783 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:05:02.785 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T01:05:02.824 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:05:03.069 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-webob. 2024-04-26T01:05:03.078 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T01:05:03.085 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T01:05:03.104 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T01:05:03.109 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T01:05:03.119 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:05:03.136 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T01:05:03.144 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:05:03.175 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T01:05:03.394 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T01:05:03.407 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T01:05:03.414 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T01:05:03.430 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:03.443 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T01:05:03.444 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:05:03.464 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:03.471 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T01:05:03.510 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T01:05:03.614 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T01:05:03.641 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T01:05:03.701 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:05:03.765 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T01:05:03.792 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T01:05:03.813 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T01:05:03.831 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:05:03.841 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T01:05:03.852 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T01:05:03.879 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:03.879 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:05:03.918 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:04.025 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T01:05:04.052 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T01:05:04.092 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:05:04.182 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T01:05:04.199 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T01:05:04.232 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:05:04.246 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T01:05:04.273 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T01:05:04.313 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:05:04.378 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-paste. 2024-04-26T01:05:04.405 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T01:05:04.451 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T01:05:04.461 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:05:04.466 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T01:05:04.493 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T01:05:04.500 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T01:05:04.510 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T01:05:04.543 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T01:05:04.574 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T01:05:04.589 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T01:05:04.641 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T01:05:04.804 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T01:05:04.815 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T01:05:04.831 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T01:05:04.832 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T01:05:04.856 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T01:05:04.871 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T01:05:04.873 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:05:04.883 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T01:05:04.947 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T01:05:04.960 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T01:05:04.987 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:05.027 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:05.159 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T01:05:05.187 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T01:05:05.190 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T01:05:05.218 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T01:05:05.234 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:05:05.257 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:05:05.292 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T01:05:05.320 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T01:05:05.358 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:05:05.447 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T01:05:05.474 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:05.513 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:05.537 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T01:05:05.565 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T01:05:05.593 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T01:05:05.604 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T01:05:05.621 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T01:05:05.660 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T01:05:05.670 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T01:05:05.698 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T01:05:05.736 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T01:05:05.982 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T01:05:05.997 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T01:05:06.010 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T01:05:06.024 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T01:05:06.031 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T01:05:06.058 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:05:06.059 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T01:05:06.063 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:05:06.089 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:05:06.100 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T01:05:06.127 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:06.166 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:06.429 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T01:05:06.456 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T01:05:06.495 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T01:05:06.743 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T01:05:06.770 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T01:05:06.771 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:06.782 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-idna. 2024-04-26T01:05:06.787 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:06.798 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T01:05:06.810 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:06.820 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:06.832 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T01:05:07.138 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T01:05:07.143 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T01:05:07.156 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T01:05:07.165 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T01:05:07.170 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T01:05:07.184 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T01:05:07.204 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:05:07.209 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:05:07.222 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:05:07.236 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph. 2024-04-26T01:05:07.263 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../066-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:07.303 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:07.490 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T01:05:07.558 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-requests. 2024-04-26T01:05:07.558 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T01:05:07.558 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:05:07.558 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:05:07.567 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T01:05:07.595 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:05:07.605 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:05:07.631 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T01:05:07.633 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:05:07.658 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:07.697 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:07.776 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package jq. 2024-04-26T01:05:07.793 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:05:07.826 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T01:05:07.888 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package jq. 2024-04-26T01:05:07.905 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:05:07.916 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T01:05:07.938 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T01:05:07.943 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T01:05:07.983 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:05:08.045 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package socat. 2024-04-26T01:05:08.061 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T01:05:08.075 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T01:05:08.102 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:08.112 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:05:08.142 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:08.191 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package socat. 2024-04-26T01:05:08.219 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T01:05:08.257 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:05:08.319 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T01:05:08.347 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:08.386 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:08.423 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T01:05:08.451 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T01:05:08.490 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:05:08.503 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package cephadm. 2024-04-26T01:05:08.518 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../069-cephadm_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:08.553 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:08.560 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T01:05:08.577 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T01:05:08.618 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:05:08.732 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T01:05:08.749 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:08.782 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:08.810 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-test. 2024-04-26T01:05:08.826 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:08.860 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:08.864 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T01:05:08.891 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T01:05:08.930 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T01:05:09.015 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-test. 2024-04-26T01:05:09.031 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:09.073 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:09.102 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T01:05:09.130 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T01:05:09.168 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:05:09.224 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T01:05:09.237 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:05:09.274 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:05:09.471 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T01:05:09.499 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T01:05:09.538 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T01:05:09.850 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T01:05:09.877 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:09.908 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:10.270 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T01:05:10.297 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:10.329 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-babel. 2024-04-26T01:05:10.344 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:10.356 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:05:10.404 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:05:10.749 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T01:05:10.776 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T01:05:10.815 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:05:11.057 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T01:05:11.084 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:11.098 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T01:05:11.126 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:11.131 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:11.143 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T01:05:11.165 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:11.169 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:11.209 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:11.410 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T01:05:11.426 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:11.427 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T01:05:11.455 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:11.468 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:11.493 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:11.554 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T01:05:11.581 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T01:05:11.629 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T01:05:11.796 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T01:05:11.797 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T01:05:11.823 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T01:05:11.825 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:11.862 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:05:11.864 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:11.915 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-routes. 2024-04-26T01:05:11.942 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T01:05:11.982 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:05:12.183 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T01:05:12.208 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T01:05:12.211 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T01:05:12.236 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T01:05:12.250 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:05:12.275 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:05:12.327 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T01:05:12.353 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:12.385 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph. 2024-04-26T01:05:12.393 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:12.401 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../066-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:12.434 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:12.587 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T01:05:12.613 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T01:05:12.614 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T01:05:12.641 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T01:05:12.687 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:05:12.704 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:05:12.721 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T01:05:12.748 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:12.787 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:13.100 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T01:05:13.125 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package pkg-config. 2024-04-26T01:05:13.128 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T01:05:13.153 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T01:05:13.166 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:05:13.192 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:05:13.350 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T01:05:13.376 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:13.416 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:13.453 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T01:05:13.481 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T01:05:13.519 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:05:13.597 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package pkg-config. 2024-04-26T01:05:13.598 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T01:05:13.614 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T01:05:13.625 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T01:05:13.656 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:05:13.663 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:05:13.802 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package cephadm. 2024-04-26T01:05:13.829 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../069-cephadm_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:13.869 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:13.883 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T01:05:13.911 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T01:05:13.941 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:05:13.958 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T01:05:13.986 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T01:05:14.024 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T01:05:14.121 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T01:05:14.137 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T01:05:14.160 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T01:05:14.180 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T01:05:14.187 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T01:05:14.226 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:05:14.327 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T01:05:14.344 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T01:05:14.354 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T01:05:14.372 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T01:05:14.393 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T01:05:14.402 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T01:05:14.483 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T01:05:14.499 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:05:14.532 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:05:14.597 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T01:05:14.623 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T01:05:14.663 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:05:14.697 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T01:05:14.725 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T01:05:14.763 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T01:05:14.897 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T01:05:14.924 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T01:05:14.940 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T01:05:14.963 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:05:14.967 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T01:05:15.007 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:05:15.083 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T01:05:15.111 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T01:05:15.149 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:05:15.250 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T01:05:15.277 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T01:05:15.316 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T01:05:15.627 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-py. 2024-04-26T01:05:15.630 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-babel. 2024-04-26T01:05:15.643 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T01:05:15.658 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:05:15.677 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T01:05:15.691 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T01:05:15.696 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:05:15.718 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:15.786 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T01:05:15.814 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T01:05:15.851 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:15.852 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T01:05:15.963 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T01:05:15.979 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T01:05:16.013 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:05:16.058 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T01:05:16.085 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T01:05:16.124 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:05:16.197 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-py. 2024-04-26T01:05:16.226 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T01:05:16.263 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T01:05:16.459 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T01:05:16.477 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T01:05:16.487 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T01:05:16.505 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:16.517 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:05:16.544 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:16.617 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T01:05:16.645 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T01:05:16.683 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:05:16.796 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T01:05:16.823 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T01:05:16.828 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-toml. 2024-04-26T01:05:16.855 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T01:05:16.862 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T01:05:16.889 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T01:05:16.894 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T01:05:16.917 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T01:05:16.956 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T01:05:17.124 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T01:05:17.137 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T01:05:17.139 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T01:05:17.165 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T01:05:17.173 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:05:17.197 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T01:05:17.203 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:05:17.225 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T01:05:17.259 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-routes. 2024-04-26T01:05:17.263 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:05:17.286 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T01:05:17.325 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:05:17.477 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T01:05:17.504 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T01:05:17.531 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-toml. 2024-04-26T01:05:17.543 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:05:17.560 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T01:05:17.598 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T01:05:17.625 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T01:05:17.653 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T01:05:17.670 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T01:05:17.692 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:05:17.698 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:17.737 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:17.821 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T01:05:17.848 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T01:05:17.888 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T01:05:17.900 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T01:05:17.929 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T01:05:17.961 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T01:05:17.967 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:05:17.988 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T01:05:18.027 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:05:18.183 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T01:05:18.199 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T01:05:18.241 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T01:05:18.312 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T01:05:18.341 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T01:05:18.379 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:05:18.561 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T01:05:18.588 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T01:05:18.627 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:05:18.631 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T01:05:18.648 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T01:05:18.690 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:05:18.815 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package radosgw. 2024-04-26T01:05:18.843 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../116-radosgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:18.881 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:18.984 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T01:05:19.000 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T01:05:19.067 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:05:19.106 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T01:05:19.133 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T01:05:19.280 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T01:05:19.443 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package radosgw. 2024-04-26T01:05:19.472 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../116-radosgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:19.510 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:19.563 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T01:05:19.575 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T01:05:19.590 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T01:05:19.602 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T01:05:19.629 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:05:19.641 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:05:19.659 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T01:05:19.675 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:19.717 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:19.991 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T01:05:20.007 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T01:05:20.025 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T01:05:20.041 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:05:20.272 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:05:20.304 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T01:05:20.310 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T01:05:20.314 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T01:05:20.321 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:20.337 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T01:05:20.342 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:20.376 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:05:20.380 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:20.390 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:05:20.622 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T01:05:20.667 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T01:05:20.668 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T01:05:20.693 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T01:05:20.725 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:05:20.926 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:05:20.984 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T01:05:21.035 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:05:21.036 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T01:05:21.064 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:05:21.102 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:05:21.113 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T01:05:21.141 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:21.256 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:21.278 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T01:05:21.388 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package jq. 2024-04-26T01:05:21.405 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:05:21.417 INFO:teuthology.orchestra.run.smithi164.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T01:05:21.447 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T01:05:21.483 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:21.643 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T01:05:21.699 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package socat. 2024-04-26T01:05:21.715 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T01:05:21.757 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:05:22.040 INFO:teuthology.orchestra.run.smithi164.stdout:Adding system user cephadm....done 2024-04-26T01:05:22.052 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T01:05:22.068 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T01:05:22.102 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:05:22.104 INFO:teuthology.orchestra.run.smithi012.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T01:05:22.180 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:22.205 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:05:22.210 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T01:05:22.228 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T01:05:22.268 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T01:05:22.457 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:05:22.473 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-test. 2024-04-26T01:05:22.501 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:22.547 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T01:05:22.564 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:22.574 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T01:05:22.613 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:05:22.691 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:05:22.711 INFO:teuthology.orchestra.run.smithi012.stdout:Adding system user cephadm....done 2024-04-26T01:05:22.817 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:05:22.835 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:05:22.934 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T01:05:22.961 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T01:05:23.000 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:05:23.076 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T01:05:23.111 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:05:23.261 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T01:05:23.289 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T01:05:23.319 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T01:05:23.328 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:05:23.362 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:05:23.478 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T01:05:23.488 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:05:23.756 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T01:05:23.763 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:05:23.865 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T01:05:23.893 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T01:05:23.931 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T01:05:24.041 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:05:24.124 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T01:05:24.192 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T01:05:24.218 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T01:05:24.235 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T01:05:24.268 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:05:24.383 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T01:05:24.502 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:05:24.513 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T01:05:24.541 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T01:05:24.579 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T01:05:24.643 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:05:24.785 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T01:05:24.812 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:24.843 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:24.879 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T01:05:24.891 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T01:05:24.894 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:05:24.919 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T01:05:24.958 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:05:25.122 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T01:05:25.138 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T01:05:25.138 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:25.153 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:05:25.172 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:25.396 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:05:25.431 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:05:25.483 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T01:05:25.510 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T01:05:25.550 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:05:25.630 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T01:05:25.647 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:25.664 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:25.674 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:05:25.680 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:25.896 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T01:05:25.923 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T01:05:25.925 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:05:25.962 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:05:25.973 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:05:26.008 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T01:05:26.036 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T01:05:26.076 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:05:26.083 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:05:26.176 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:05:26.184 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:05:26.310 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:05:26.316 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T01:05:26.343 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T01:05:26.382 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:05:26.411 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T01:05:26.427 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:26.439 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:05:26.478 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:05:26.569 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:05:26.714 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package jq. 2024-04-26T01:05:26.730 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:05:26.764 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T01:05:26.813 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package pkg-config. 2024-04-26T01:05:26.820 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:05:26.840 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T01:05:26.879 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:05:26.946 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:05:26.952 INFO:teuthology.orchestra.run.smithi164.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T01:05:26.975 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package socat. 2024-04-26T01:05:26.991 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T01:05:27.009 INFO:teuthology.orchestra.run.smithi164.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T01:05:27.034 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:05:27.072 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:05:27.073 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:05:27.115 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T01:05:27.131 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T01:05:27.165 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:05:27.198 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:05:27.199 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T01:05:27.353 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T01:05:27.378 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T01:05:27.420 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:05:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:05:27.541 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:05:27.576 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T01:05:27.592 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T01:05:27.626 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T01:05:27.650 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T01:05:27.799 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-test. 2024-04-26T01:05:27.827 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:27.837 INFO:teuthology.orchestra.run.smithi012.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T01:05:27.845 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T01:05:27.861 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T01:05:27.866 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:27.872 INFO:teuthology.orchestra.run.smithi012.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T01:05:27.895 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T01:05:27.918 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:05:27.936 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:05:28.095 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T01:05:28.178 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:05:28.199 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T01:05:28.227 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T01:05:28.265 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:05:28.463 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:05:28.512 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T01:05:28.577 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T01:05:28.588 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T01:05:28.605 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T01:05:28.621 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:05:28.643 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T01:05:28.865 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:05:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-py. 2024-04-26T01:05:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T01:05:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T01:05:29.149 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:05:29.248 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:05:29.357 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T01:05:29.385 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T01:05:29.424 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:05:29.459 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T01:05:29.482 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:05:29.609 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:05:29.610 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:05:29.734 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:05:29.986 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T01:05:30.014 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T01:05:30.052 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:05:30.229 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T01:05:30.237 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T01:05:30.262 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:05:30.295 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:30.305 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-toml. 2024-04-26T01:05:30.328 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:05:30.333 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T01:05:30.372 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T01:05:30.588 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:05:30.591 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T01:05:30.619 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:30.633 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T01:05:30.657 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:30.660 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T01:05:30.699 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:05:30.714 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:05:30.770 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T01:05:30.834 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:05:30.944 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T01:05:30.972 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T01:05:31.010 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:05:31.013 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:05:31.019 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T01:05:31.047 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T01:05:31.077 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:05:31.321 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T01:05:31.337 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T01:05:31.380 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:05:31.381 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T01:05:31.390 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:05:31.410 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T01:05:31.424 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T01:05:31.448 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:05:31.674 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:05:31.811 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T01:05:31.817 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T01:05:31.839 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T01:05:31.877 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:05:32.000 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:05:32.093 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:05:32.109 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package radosgw. 2024-04-26T01:05:32.137 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../116-radosgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:32.175 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:32.252 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:05:32.266 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package pkg-config. 2024-04-26T01:05:32.294 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T01:05:32.324 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:05:32.377 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:32.461 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:05:32.544 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T01:05:32.572 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T01:05:32.610 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:05:32.662 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T01:05:32.762 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:05:32.970 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T01:05:32.980 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:05:32.989 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T01:05:32.997 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:33.017 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T01:05:33.036 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:33.055 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T01:05:33.114 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:05:33.356 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:05:33.358 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T01:05:33.359 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T01:05:33.386 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T01:05:33.424 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T01:05:33.482 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:33.618 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:05:33.727 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:05:33.745 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T01:05:33.773 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T01:05:33.775 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T01:05:33.811 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:05:33.962 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T01:05:34.101 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:05:34.106 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T01:05:34.134 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T01:05:34.172 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T01:05:34.176 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:05:34.258 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T01:05:34.428 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:05:34.525 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-py. 2024-04-26T01:05:34.537 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T01:05:34.553 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T01:05:34.609 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T01:05:34.691 INFO:teuthology.orchestra.run.smithi116.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T01:05:34.755 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:34.789 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:05:34.890 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:05:34.937 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T01:05:34.965 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T01:05:34.991 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:05:35.003 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:05:35.280 INFO:teuthology.orchestra.run.smithi116.stdout:Adding system user cephadm....done 2024-04-26T01:05:35.283 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:05:35.348 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:05:35.436 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:05:35.466 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T01:05:35.494 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T01:05:35.532 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:05:35.535 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T01:05:35.575 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:05:35.695 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:05:35.703 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T01:05:35.819 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:05:35.827 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-toml. 2024-04-26T01:05:35.855 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T01:05:35.894 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T01:05:35.937 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:05:35.946 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:05:35.977 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:05:36.072 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:05:36.103 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:05:36.146 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T01:05:36.163 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T01:05:36.196 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:05:36.222 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:05:36.246 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:05:36.340 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T01:05:36.448 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T01:05:36.533 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T01:05:36.547 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:05:36.562 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T01:05:36.591 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:05:36.600 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:05:36.682 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T01:05:36.709 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T01:05:36.798 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T01:05:36.869 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T01:05:36.886 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T01:05:36.917 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:05:36.928 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:05:37.019 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:05:37.049 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:05:37.159 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T01:05:37.175 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:05:37.363 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T01:05:37.419 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:05:37.494 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:05:37.536 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T01:05:37.614 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T01:05:37.698 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package radosgw. 2024-04-26T01:05:37.727 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../116-radosgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:37.753 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T01:05:37.765 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:37.804 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:05:37.881 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:05:37.914 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:05:38.021 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T01:05:38.118 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:05:38.245 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:05:38.271 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:05:38.376 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:05:38.508 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:05:38.522 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T01:05:38.626 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T01:05:38.627 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:05:38.643 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:05:38.676 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:38.758 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:05:38.782 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:05:38.861 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:38.884 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:05:38.908 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T01:05:38.962 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T01:05:39.010 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:05:39.098 INFO:teuthology.orchestra.run.smithi164.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T01:05:39.121 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:05:39.193 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:05:39.327 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:05:39.328 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:05:39.397 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:05:39.438 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:05:39.506 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:05:39.523 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:05:39.619 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:05:39.624 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:05:39.657 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:05:39.806 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:05:39.877 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T01:05:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:05:39.929 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:05:40.049 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T01:05:40.171 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:05:40.175 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T01:05:40.215 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T01:05:40.265 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T01:05:40.297 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T01:05:40.297 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:05:40.361 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:05:40.415 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:05:40.434 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:05:40.488 INFO:teuthology.orchestra.run.smithi012.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T01:05:40.504 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T01:05:40.585 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:05:40.671 INFO:teuthology.orchestra.run.smithi008.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T01:05:40.734 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:05:40.768 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:40.859 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:05:40.863 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:05:40.903 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:05:40.989 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T01:05:41.187 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:05:41.249 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:05:41.278 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:05:41.463 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T01:05:41.492 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:05:41.579 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T01:05:41.589 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T01:05:41.660 INFO:teuthology.orchestra.run.smithi008.stdout:Adding system user cephadm....done 2024-04-26T01:05:41.705 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T01:05:41.783 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:05:41.801 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T01:05:41.840 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T01:05:41.910 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:05:41.973 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:05:42.077 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:05:42.082 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:05:42.100 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:05:42.336 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:05:42.399 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:05:42.453 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:05:42.462 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:05:42.579 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:05:42.709 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:05:42.722 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T01:05:42.830 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T01:05:42.830 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:05:42.956 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:05:42.981 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:05:43.011 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T01:05:43.090 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:05:43.099 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T01:05:43.106 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:05:43.136 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T01:05:43.383 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T01:05:43.392 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:05:43.413 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T01:05:43.425 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:05:43.551 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:43.677 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:43.697 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:05:43.753 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T01:05:43.768 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T01:05:43.795 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:05:44.021 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T01:05:44.041 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:05:44.043 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:05:44.106 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:44.232 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:44.289 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:05:44.358 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T01:05:44.412 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:05:44.460 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T01:05:44.541 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:05:44.643 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:44.688 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:05:44.754 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:05:44.769 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:44.800 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:05:44.911 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:45.014 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:05:45.060 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:05:45.089 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:05:45.125 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:05:45.125 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:05:45.198 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:45.266 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:05:45.299 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:45.311 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:45.392 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:45.409 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:05:45.579 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:05:45.596 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:45.693 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T01:05:45.704 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:05:45.753 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:45.831 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:05:45.880 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:45.905 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:45.957 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:05:46.002 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:05:46.005 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T01:05:46.031 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:46.157 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:05:46.216 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:05:46.282 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:46.408 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:46.572 INFO:teuthology.orchestra.run.smithi008.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T01:05:46.606 INFO:teuthology.orchestra.run.smithi008.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T01:05:46.670 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:05:46.717 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:46.838 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T01:05:46.937 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:05:46.937 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:05:47.108 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:05:47.205 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:05:47.331 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T01:05:47.359 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:05:47.402 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:47.435 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:47.493 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T01:05:47.520 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:47.553 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:47.591 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:05:47.646 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:47.654 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:47.753 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:05:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:05:47.772 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:47.851 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:05:47.879 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:05:47.898 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:48.005 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:05:48.152 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T01:05:48.253 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:05:48.281 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:05:48.448 INFO:teuthology.orchestra.run.smithi164.stdout:Adding group ceph....done 2024-04-26T01:05:48.541 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T01:05:48.783 INFO:teuthology.orchestra.run.smithi164.stdout:Adding system user ceph....done 2024-04-26T01:05:48.791 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:05:48.910 INFO:teuthology.orchestra.run.smithi164.stdout:Setting system user ceph properties....done 2024-04-26T01:05:48.918 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:05:48.922 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:05:48.927 INFO:teuthology.orchestra.run.smithi164.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T01:05:48.992 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:49.068 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T01:05:49.118 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:49.181 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:05:49.227 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:05:49.244 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:49.291 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:05:49.370 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:49.390 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T01:05:49.391 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:05:49.487 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T01:05:49.496 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:49.701 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T01:05:49.729 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T01:05:49.816 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:49.941 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:49.980 INFO:teuthology.orchestra.run.smithi012.stdout:Adding group ceph....done 2024-04-26T01:05:50.094 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T01:05:50.289 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:05:50.337 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:05:50.358 INFO:teuthology.orchestra.run.smithi012.stdout:Adding system user ceph....done 2024-04-26T01:05:50.415 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:05:50.415 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:05:50.526 INFO:teuthology.orchestra.run.smithi012.stdout:Setting system user ceph properties....done 2024-04-26T01:05:50.540 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T01:05:50.541 INFO:teuthology.orchestra.run.smithi012.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T01:05:50.693 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T01:05:50.722 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:05:50.807 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:05:50.877 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:50.934 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T01:05:51.023 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T01:05:51.023 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:05:51.206 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T01:05:51.235 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:05:51.361 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:05:51.375 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:05:51.522 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:51.631 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:51.634 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:05:51.645 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:05:51.662 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:51.736 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:51.869 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:05:51.869 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:05:51.938 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:05:52.029 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T01:05:52.095 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:05:52.095 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:05:52.172 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:05:52.307 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:05:52.347 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:52.355 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:05:52.433 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:05:52.523 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T01:05:52.567 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:05:52.568 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:05:52.591 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:52.759 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:05:52.885 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:05:52.966 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T01:05:53.006 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:53.279 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:05:53.286 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:05:53.286 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:05:53.477 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:53.521 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T01:05:53.552 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:05:53.647 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T01:05:53.702 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:05:53.702 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:05:53.774 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:53.796 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:05:53.898 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:05:53.902 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:53.930 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T01:05:54.024 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:05:54.043 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:54.195 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:54.206 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:05:54.254 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:05:54.254 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:05:54.333 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:05:54.342 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:05:54.423 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:05:54.423 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:05:54.484 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:05:54.643 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:05:54.686 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:54.820 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:54.919 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T01:05:54.921 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:54.929 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:54.944 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:05:55.045 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T01:05:55.072 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:55.186 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T01:05:55.195 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:05:55.195 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:05:55.304 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T01:05:55.418 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T01:05:55.438 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:05:55.517 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T01:05:55.564 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:05:55.580 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:05:55.689 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:55.794 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T01:05:55.824 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:55.873 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:05:55.926 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:05:55.974 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:56.133 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T01:05:56.193 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:05:56.193 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:05:56.327 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T01:05:56.393 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T01:05:56.604 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:05:56.636 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:05:56.701 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:56.852 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:56.870 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T01:05:56.929 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T01:05:56.939 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:05:57.005 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:57.066 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:57.138 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:05:57.147 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:57.192 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:57.264 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T01:05:57.318 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:05:57.484 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T01:05:57.573 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:05:57.601 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T01:05:57.620 INFO:teuthology.orchestra.run.smithi164.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T01:05:57.626 DEBUG:teuthology.parallel:result is None 2024-04-26T01:05:57.637 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:57.699 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:05:57.738 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:57.753 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T01:05:57.864 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T01:05:58.009 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:05:58.149 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:58.275 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:58.285 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:05:58.426 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:58.545 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:05:58.636 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:05:58.636 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:05:58.671 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:05:58.797 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:05:58.879 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T01:05:58.885 INFO:teuthology.orchestra.run.smithi008.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T01:05:59.115 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:05:59.119 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:59.228 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:59.241 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:05:59.338 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:05:59.447 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:05:59.529 INFO:teuthology.orchestra.run.smithi012.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T01:05:59.534 DEBUG:teuthology.parallel:result is None 2024-04-26T01:05:59.668 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:05:59.928 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T01:06:00.037 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T01:06:00.338 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:06:00.464 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:06:00.684 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:00.799 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:06:00.810 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:00.936 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:01.062 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:01.100 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:06:01.179 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:01.384 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T01:06:01.486 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T01:06:01.689 INFO:teuthology.orchestra.run.smithi116.stdout:Adding group ceph....done 2024-04-26T01:06:01.762 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T01:06:02.038 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:06:02.049 INFO:teuthology.orchestra.run.smithi116.stdout:Adding system user ceph....done 2024-04-26T01:06:02.209 INFO:teuthology.orchestra.run.smithi116.stdout:Setting system user ceph properties....done 2024-04-26T01:06:02.223 INFO:teuthology.orchestra.run.smithi116.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T01:06:02.356 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T01:06:02.417 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:06:02.660 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T01:06:02.827 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T01:06:03.103 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:06:03.105 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:03.207 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:03.448 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:06:03.574 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:03.675 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:06:03.675 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:06:03.700 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:03.826 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:06:04.137 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:04.184 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:04.263 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:04.389 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T01:06:04.666 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:04.745 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T01:06:04.792 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:04.910 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:05.113 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:06:05.113 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:06:05.219 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:05.431 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:06:05.431 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:06:05.578 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:05.704 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:05.830 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:05.916 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:05.981 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:06:06.130 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:06:06.130 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:06:06.617 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:06.866 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:06:06.866 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:06:07.254 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:07.368 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:07.379 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:07.478 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:07.505 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:07.629 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:07.631 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:07.758 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:07.842 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:06:07.842 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:06:08.259 INFO:teuthology.orchestra.run.smithi008.stdout:Adding group ceph....done 2024-04-26T01:06:08.297 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:08.448 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:08.574 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:08.603 INFO:teuthology.orchestra.run.smithi008.stdout:Adding system user ceph....done 2024-04-26T01:06:08.725 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:08.771 INFO:teuthology.orchestra.run.smithi008.stdout:Setting system user ceph properties....done 2024-04-26T01:06:08.784 INFO:teuthology.orchestra.run.smithi008.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T01:06:08.929 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T01:06:09.126 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T01:06:09.220 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T01:06:09.260 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T01:06:09.364 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T01:06:09.768 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:09.869 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:10.312 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:06:10.313 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:06:10.473 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T01:06:10.813 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:11.174 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T01:06:11.285 INFO:teuthology.orchestra.run.smithi116.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T01:06:11.291 DEBUG:teuthology.parallel:result is None 2024-04-26T01:06:11.665 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:11.885 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:06:11.886 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:06:12.399 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:12.628 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:06:12.628 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:06:13.092 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:13.349 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:06:13.349 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:06:13.836 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:13.962 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:14.105 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:14.306 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:06:14.307 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:06:14.789 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:14.957 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:15.083 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:15.454 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:06:15.940 INFO:teuthology.orchestra.run.smithi008.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T01:06:16.049 INFO:teuthology.orchestra.run.smithi008.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T01:06:16.193 INFO:teuthology.orchestra.run.smithi008.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T01:06:17.236 INFO:teuthology.orchestra.run.smithi008.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T01:06:17.913 INFO:teuthology.orchestra.run.smithi008.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T01:06:17.918 DEBUG:teuthology.parallel:result is None 2024-04-26T01:06:17.918 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:06:18.077 DEBUG:teuthology.orchestra.run.smithi008:> dpkg-query -W -f '${Version}' ceph 2024-04-26T01:06:18.100 INFO:teuthology.orchestra.run.smithi008.stdout:19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:06:18.100 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:06:18.100 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445-gd2c45d97-1jammy is installed. 2024-04-26T01:06:18.102 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:06:18.388 DEBUG:teuthology.orchestra.run.smithi012:> dpkg-query -W -f '${Version}' ceph 2024-04-26T01:06:18.408 INFO:teuthology.orchestra.run.smithi012.stdout:19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:06:18.409 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:06:18.409 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445-gd2c45d97-1jammy is installed. 2024-04-26T01:06:18.411 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:06:18.560 DEBUG:teuthology.orchestra.run.smithi116:> dpkg-query -W -f '${Version}' ceph 2024-04-26T01:06:18.579 INFO:teuthology.orchestra.run.smithi116.stdout:19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:06:18.579 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:06:18.579 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445-gd2c45d97-1jammy is installed. 2024-04-26T01:06:18.581 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:06:18.731 DEBUG:teuthology.orchestra.run.smithi164:> dpkg-query -W -f '${Version}' ceph 2024-04-26T01:06:18.750 INFO:teuthology.orchestra.run.smithi164.stdout:19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:06:18.751 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:06:18.751 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445-gd2c45d97-1jammy is installed. 2024-04-26T01:06:18.752 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T01:06:18.753 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:06:18.753 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:06:18.767 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:06:18.768 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:06:18.783 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:06:18.783 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:06:18.797 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:06:18.797 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:06:18.811 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T01:06:18.811 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:06:18.811 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:06:18.825 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:06:18.880 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:06:18.881 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:06:18.895 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:06:18.951 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:06:18.951 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:06:18.966 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:06:19.022 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:06:19.023 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:06:19.036 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:06:19.090 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T01:06:19.091 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:06:19.091 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:06:19.106 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:06:19.159 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:06:19.159 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:06:19.173 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:06:19.226 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:06:19.226 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:06:19.240 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:06:19.294 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:06:19.294 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:06:19.311 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:06:19.367 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T01:06:19.367 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:06:19.367 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:06:19.381 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:06:19.435 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:06:19.435 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:06:19.449 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:06:19.502 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:06:19.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:06:19.515 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:06:19.569 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:06:19.569 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:06:19.583 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:06:19.639 INFO:teuthology.run_tasks:Running task ceph... 2024-04-26T01:06:19.738 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-26T01:06:19.739 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 777 /var/log/ceph 2024-04-26T01:06:19.741 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /var/log/ceph 2024-04-26T01:06:19.743 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /var/log/ceph 2024-04-26T01:06:19.746 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /var/log/ceph 2024-04-26T01:06:19.759 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-26T01:06:19.759 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T01:06:19.796 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T01:06:19.800 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T01:06:19.802 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T01:06:19.814 INFO:tasks.ceph:Creating extra log directories... 2024-04-26T01:06:19.814 DEBUG:teuthology.orchestra.run.smithi008:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T01:06:19.853 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T01:06:19.856 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T01:06:19.858 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T01:06:19.885 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-26T01:06:19.885 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-26T01:06:19.885 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968', 'branch': 'squid', 'description': 'rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7673968', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 9886, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'sleep_before_teardown': 0, 'subset': '30270/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a', 'targets': {'smithi008.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK4frcFiXxuvFihce9UuoZoisKkhLyfeOBzqyu9QD0Ag0Ib9TqpDIqR2y/Kw1qisy2aNXGlAAJnadpePsnNBvdA=', 'smithi012.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJv4HnRZVX1vvurODkhJ8tjR5Yt3zLZhRKjDfe5yPtS6hL4vO6RwTBiRjDbxfjdXiolat12eFeWP8Z74UFf2nC4=', 'smithi116.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJspkZF0hdTqWWhaEn4cSjWa0GPSlwvhE+s0JRCJAovLtBTD51U4AfMgOb3dloOtVSQDLdhnXBKP6bI3g/jf4yQ=', 'smithi164.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNQJPJnLOnvuCUj2BVAR4HkQSosOA0k3Q9YV8c3hcP5RMq297jKmS9Ae/e/iHxYNT0dpkdDmJXfeHeIE+YweZGo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'cluster': 'ceph'}}, {'thrashosds': {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-25_21:00:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-26T01:06:19.886 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T01:06:19.909 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T01:06:19.912 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T01:06:19.915 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T01:06:19.921 DEBUG:teuthology.orchestra.run.smithi008:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T01:06:19.957 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T01:06:19.964 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T01:06:19.966 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T01:06:19.979 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:06:19.979 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:06:20.016 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:06:20.016 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_1 2024-04-26T01:06:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:06:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:20.064 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 876 Links: 1 2024-04-26T01:06:20.064 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:20.064 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-04-26 01:00:22.455816536 +0000 2024-04-26T01:06:20.064 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-04-26 01:00:22.175819037 +0000 2024-04-26T01:06:20.064 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-04-26 01:00:22.175819037 +0000 2024-04-26T01:06:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: - 2024-04-26T01:06:20.065 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:06:20.118 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-04-26T01:06:20.118 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-04-26T01:06:20.118 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000321785 s, 1.6 MB/s 2024-04-26T01:06:20.120 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:06:20.169 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_2 2024-04-26T01:06:20.216 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:06:20.216 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:20.217 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 894 Links: 1 2024-04-26T01:06:20.217 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:20.217 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-04-26 01:00:22.951812106 +0000 2024-04-26T01:06:20.217 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-04-26 01:00:22.703814321 +0000 2024-04-26T01:06:20.217 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-04-26 01:00:22.703814321 +0000 2024-04-26T01:06:20.217 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: - 2024-04-26T01:06:20.217 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:06:20.274 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-04-26T01:06:20.274 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-04-26T01:06:20.274 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000232656 s, 2.2 MB/s 2024-04-26T01:06:20.276 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:06:20.326 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_3 2024-04-26T01:06:20.372 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:06:20.373 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:20.373 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 904 Links: 1 2024-04-26T01:06:20.373 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:20.373 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-04-26 01:00:23.447807675 +0000 2024-04-26T01:06:20.373 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-04-26 01:00:23.183810033 +0000 2024-04-26T01:06:20.373 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-04-26 01:00:23.183810033 +0000 2024-04-26T01:06:20.373 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: - 2024-04-26T01:06:20.373 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:06:20.425 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-04-26T01:06:20.426 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-04-26T01:06:20.426 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000210307 s, 2.4 MB/s 2024-04-26T01:06:20.427 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:06:20.478 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_4 2024-04-26T01:06:20.524 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:06:20.524 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:20.524 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 927 Links: 1 2024-04-26T01:06:20.524 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:20.524 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-04-26 01:00:23.687805531 +0000 2024-04-26T01:06:20.524 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-04-26 01:00:23.683805567 +0000 2024-04-26T01:06:20.524 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-04-26 01:00:23.683805567 +0000 2024-04-26T01:06:20.525 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: - 2024-04-26T01:06:20.525 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:06:20.579 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-04-26T01:06:20.579 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-04-26T01:06:20.579 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000296661 s, 1.7 MB/s 2024-04-26T01:06:20.581 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:06:20.629 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:20.629 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:06:20.629 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:06:20.636 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:06:20.636 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 2024-04-26T01:06:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:06:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:20.684 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 889 Links: 1 2024-04-26T01:06:20.684 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:20.684 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-26 01:00:40.581731895 +0000 2024-04-26T01:06:20.684 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-26 01:00:40.297734432 +0000 2024-04-26T01:06:20.684 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-26 01:00:40.297734432 +0000 2024-04-26T01:06:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2024-04-26T01:06:20.685 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:06:20.738 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-26T01:06:20.738 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-26T01:06:20.739 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000375425 s, 1.4 MB/s 2024-04-26T01:06:20.740 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:06:20.790 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 2024-04-26T01:06:20.835 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:06:20.836 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:20.836 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 905 Links: 1 2024-04-26T01:06:20.836 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:20.836 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-26 01:00:41.093727322 +0000 2024-04-26T01:06:20.836 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-26 01:00:40.797729965 +0000 2024-04-26T01:06:20.836 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-26 01:00:40.797729965 +0000 2024-04-26T01:06:20.836 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2024-04-26T01:06:20.836 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:06:20.890 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-26T01:06:20.890 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-26T01:06:20.890 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000348075 s, 1.5 MB/s 2024-04-26T01:06:20.891 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:06:20.940 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 2024-04-26T01:06:20.988 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:06:20.988 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:20.988 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 921 Links: 1 2024-04-26T01:06:20.988 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:20.988 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-26 01:00:41.621722605 +0000 2024-04-26T01:06:20.988 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-26 01:00:41.345725071 +0000 2024-04-26T01:06:20.988 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-26 01:00:41.345725071 +0000 2024-04-26T01:06:20.988 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2024-04-26T01:06:20.989 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:06:21.042 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-26T01:06:21.042 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-26T01:06:21.042 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000425804 s, 1.2 MB/s 2024-04-26T01:06:21.044 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:06:21.093 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 2024-04-26T01:06:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:06:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:21.139 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 938 Links: 1 2024-04-26T01:06:21.139 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:21.139 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-26 01:00:42.093718388 +0000 2024-04-26T01:06:21.139 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-26 01:00:41.857720497 +0000 2024-04-26T01:06:21.139 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-26 01:00:41.857720497 +0000 2024-04-26T01:06:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2024-04-26T01:06:21.140 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:06:21.194 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-26T01:06:21.194 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-26T01:06:21.194 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000387847 s, 1.3 MB/s 2024-04-26T01:06:21.195 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:06:21.245 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:21.245 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:06:21.245 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:06:21.251 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:06:21.251 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-04-26T01:06:21.298 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:06:21.299 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:21.299 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 881 Links: 1 2024-04-26T01:06:21.299 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:21.299 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-04-26 01:00:19.634988315 +0000 2024-04-26T01:06:21.299 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-04-26 01:00:19.630988351 +0000 2024-04-26T01:06:21.299 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-04-26 01:00:19.630988351 +0000 2024-04-26T01:06:21.299 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2024-04-26T01:06:21.299 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:06:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-04-26T01:06:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-04-26T01:06:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000399934 s, 1.3 MB/s 2024-04-26T01:06:21.354 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:06:21.404 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-04-26T01:06:21.450 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:06:21.450 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:21.451 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-04-26T01:06:21.451 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:21.451 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-04-26 01:00:20.450981026 +0000 2024-04-26T01:06:21.451 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-04-26 01:00:20.166983564 +0000 2024-04-26T01:06:21.451 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-04-26 01:00:20.166983564 +0000 2024-04-26T01:06:21.451 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2024-04-26T01:06:21.451 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:06:21.505 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-04-26T01:06:21.505 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-04-26T01:06:21.505 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.00044185 s, 1.2 MB/s 2024-04-26T01:06:21.506 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:06:21.555 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-04-26T01:06:21.602 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:06:21.602 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:21.602 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 914 Links: 1 2024-04-26T01:06:21.603 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:21.603 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-04-26 01:00:20.938976668 +0000 2024-04-26T01:06:21.603 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-04-26 01:00:20.686978918 +0000 2024-04-26T01:06:21.603 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-04-26 01:00:20.686978918 +0000 2024-04-26T01:06:21.603 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2024-04-26T01:06:21.603 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:06:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-04-26T01:06:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-04-26T01:06:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000411603 s, 1.2 MB/s 2024-04-26T01:06:21.657 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:06:21.707 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-04-26T01:06:21.754 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:06:21.754 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:21.754 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 932 Links: 1 2024-04-26T01:06:21.754 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:21.754 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-04-26 01:00:21.478971844 +0000 2024-04-26T01:06:21.755 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-04-26 01:00:21.186974452 +0000 2024-04-26T01:06:21.755 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-04-26 01:00:21.186974452 +0000 2024-04-26T01:06:21.755 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2024-04-26T01:06:21.755 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:06:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-04-26T01:06:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-04-26T01:06:21.809 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000387637 s, 1.3 MB/s 2024-04-26T01:06:21.810 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:06:21.860 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:21.860 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:06:21.860 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:06:21.866 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:06:21.866 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 2024-04-26T01:06:21.912 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:06:21.912 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:21.912 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 807 Links: 1 2024-04-26T01:06:21.912 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:21.912 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-04-26 01:00:15.123891904 +0000 2024-04-26T01:06:21.912 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-04-26 01:00:14.863894226 +0000 2024-04-26T01:06:21.913 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-04-26 01:00:14.863894226 +0000 2024-04-26T01:06:21.913 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2024-04-26T01:06:21.913 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:06:21.966 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-04-26T01:06:21.966 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-04-26T01:06:21.966 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.00041965 s, 1.2 MB/s 2024-04-26T01:06:21.968 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:06:22.017 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 2024-04-26T01:06:22.063 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:06:22.064 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:22.064 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 827 Links: 1 2024-04-26T01:06:22.064 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:22.064 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-04-26 01:00:15.603887616 +0000 2024-04-26T01:06:22.064 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-04-26 01:00:15.339889974 +0000 2024-04-26T01:06:22.064 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-04-26 01:00:15.339889974 +0000 2024-04-26T01:06:22.064 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2024-04-26T01:06:22.064 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:06:22.117 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-04-26T01:06:22.118 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-04-26T01:06:22.118 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000414027 s, 1.2 MB/s 2024-04-26T01:06:22.119 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:06:22.169 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 2024-04-26T01:06:22.216 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:06:22.216 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:22.216 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 846 Links: 1 2024-04-26T01:06:22.216 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:22.216 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-04-26 01:00:16.103883150 +0000 2024-04-26T01:06:22.216 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-04-26 01:00:15.847885437 +0000 2024-04-26T01:06:22.216 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-04-26 01:00:15.847885437 +0000 2024-04-26T01:06:22.216 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2024-04-26T01:06:22.217 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:06:22.270 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-04-26T01:06:22.270 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-04-26T01:06:22.270 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000435749 s, 1.2 MB/s 2024-04-26T01:06:22.271 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:06:22.321 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 2024-04-26T01:06:22.367 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:06:22.368 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:06:22.368 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 864 Links: 1 2024-04-26T01:06:22.368 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:06:22.368 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-04-26 01:00:16.595878756 +0000 2024-04-26T01:06:22.368 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-04-26 01:00:16.323881185 +0000 2024-04-26T01:06:22.368 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-04-26 01:00:16.323881185 +0000 2024-04-26T01:06:22.368 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2024-04-26T01:06:22.368 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:06:22.421 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-04-26T01:06:22.421 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-04-26T01:06:22.421 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000355509 s, 1.4 MB/s 2024-04-26T01:06:22.423 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:06:22.473 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:22.473 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi008.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-04-26T01:06:22.473 INFO:tasks.ceph:Generating config... 2024-04-26T01:06:22.474 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-26T01:06:22.474 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-26T01:06:22.474 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-04-26T01:06:22.474 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-26T01:06:22.474 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-26T01:06:22.474 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-26T01:06:22.474 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-26T01:06:22.474 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-26T01:06:22.475 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-26T01:06:22.475 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-26T01:06:22.475 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-26T01:06:22.475 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-26T01:06:22.475 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-26T01:06:22.475 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-26T01:06:22.475 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-26T01:06:22.475 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.1 2024-04-26T01:06:22.475 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-26T01:06:22.475 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-26T01:06:22.475 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-26T01:06:22.475 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-04-26T01:06:22.475 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-26T01:06:22.475 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-26T01:06:22.476 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-26T01:06:22.476 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-26T01:06:22.476 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-26T01:06:22.476 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-26T01:06:22.476 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-04-26T01:06:22.476 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-04-26T01:06:22.476 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-26T01:06:22.476 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-26T01:06:22.476 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-04-26T01:06:22.476 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-04-26T01:06:22.476 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-04-26T01:06:22.476 INFO:tasks.ceph:[global] ms inject delay type = osd 2024-04-26T01:06:22.477 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-04-26T01:06:22.477 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-04-26T01:06:22.477 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-04-26T01:06:22.477 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-26T01:06:22.477 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-26T01:06:22.477 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-26T01:06:22.477 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-26T01:06:22.477 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-26T01:06:22.477 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-26T01:06:22.477 INFO:tasks.ceph:Setting up mon.a... 2024-04-26T01:06:22.477 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-26T01:06:22.511 INFO:teuthology.orchestra.run.smithi008.stdout:creating /etc/ceph/ceph.keyring 2024-04-26T01:06:22.529 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-26T01:06:22.607 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T01:06:22.664 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.8'), ('mon.b', '172.21.15.12'), ('mon.c', '172.21.15.116')] 2024-04-26T01:06:22.664 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.8,172.21.15.12,172.21.15.116', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-04-26T01:06:22.665 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:06:22.665 DEBUG:teuthology.orchestra.run.smithi008:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T01:06:22.712 DEBUG:teuthology.orchestra.run.smithi008:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.8 --add b 172.21.15.12 --add c 172.21.15.116 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:22.799 INFO:teuthology.orchestra.run.smithi008.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:22.800 INFO:teuthology.orchestra.run.smithi008.stdout:monmaptool: generated fsid 4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554 2024-04-26T01:06:22.800 INFO:teuthology.orchestra.run.smithi008.stdout:setting min_mon_release = quincy 2024-04-26T01:06:22.800 INFO:teuthology.orchestra.run.smithi008.stdout:epoch 0 2024-04-26T01:06:22.800 INFO:teuthology.orchestra.run.smithi008.stdout:fsid 4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554 2024-04-26T01:06:22.800 INFO:teuthology.orchestra.run.smithi008.stdout:last_changed 2024-04-26T01:06:22.798258+0000 2024-04-26T01:06:22.800 INFO:teuthology.orchestra.run.smithi008.stdout:created 2024-04-26T01:06:22.798258+0000 2024-04-26T01:06:22.800 INFO:teuthology.orchestra.run.smithi008.stdout:min_mon_release 17 (quincy) 2024-04-26T01:06:22.800 INFO:teuthology.orchestra.run.smithi008.stdout:election_strategy: 1 2024-04-26T01:06:22.800 INFO:teuthology.orchestra.run.smithi008.stdout:0: [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon.a 2024-04-26T01:06:22.800 INFO:teuthology.orchestra.run.smithi008.stdout:1: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.b 2024-04-26T01:06:22.800 INFO:teuthology.orchestra.run.smithi008.stdout:2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.c 2024-04-26T01:06:22.800 INFO:teuthology.orchestra.run.smithi008.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-26T01:06:22.801 DEBUG:teuthology.orchestra.run.smithi008:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T01:06:22.806 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554... 2024-04-26T01:06:22.807 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T01:06:22.853 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T01:06:22.855 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T01:06:22.857 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T01:06:22.882 INFO:teuthology.orchestra.run.smithi008.stdout:[global] 2024-04-26T01:06:22.882 INFO:teuthology.orchestra.run.smithi008.stdout: chdir = "" 2024-04-26T01:06:22.882 INFO:teuthology.orchestra.run.smithi008.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T01:06:22.882 INFO:teuthology.orchestra.run.smithi008.stdout: auth supported = cephx 2024-04-26T01:06:22.882 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.883 INFO:teuthology.orchestra.run.smithi008.stdout: filestore xattr use omap = true 2024-04-26T01:06:22.883 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.883 INFO:teuthology.orchestra.run.smithi008.stdout: mon clock drift allowed = 1.000 2024-04-26T01:06:22.883 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.883 INFO:teuthology.orchestra.run.smithi008.stdout: osd crush chooseleaf type = 0 2024-04-26T01:06:22.883 INFO:teuthology.orchestra.run.smithi008.stdout: auth debug = true 2024-04-26T01:06:22.883 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.884 INFO:teuthology.orchestra.run.smithi008.stdout: ms die on old message = true 2024-04-26T01:06:22.884 INFO:teuthology.orchestra.run.smithi008.stdout: ms die on bug = true 2024-04-26T01:06:22.884 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.884 INFO:teuthology.orchestra.run.smithi008.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T01:06:22.884 INFO:teuthology.orchestra.run.smithi008.stdout: mon pg warn max object skew = 0 2024-04-26T01:06:22.884 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.885 INFO:teuthology.orchestra.run.smithi008.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T01:06:22.885 INFO:teuthology.orchestra.run.smithi008.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T01:06:22.885 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.885 INFO:teuthology.orchestra.run.smithi008.stdout: osd pool default size = 2 2024-04-26T01:06:22.885 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.885 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd allow primary affinity = true 2024-04-26T01:06:22.885 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd allow pg remap = true 2024-04-26T01:06:22.886 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on legacy crush tunables = false 2024-04-26T01:06:22.886 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on crush straw calc version zero = false 2024-04-26T01:06:22.886 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on no sortbitwise = false 2024-04-26T01:06:22.886 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on osd down out interval zero = false 2024-04-26T01:06:22.886 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on too few osds = false 2024-04-26T01:06:22.886 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T01:06:22.887 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T01:06:22.887 INFO:teuthology.orchestra.run.smithi008.stdout: mon_allow_pool_size_one = true 2024-04-26T01:06:22.887 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.887 INFO:teuthology.orchestra.run.smithi008.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T01:06:22.887 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.887 INFO:teuthology.orchestra.run.smithi008.stdout: osd default data pool replay window = 5 2024-04-26T01:06:22.887 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.888 INFO:teuthology.orchestra.run.smithi008.stdout: mon allow pool delete = true 2024-04-26T01:06:22.888 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.888 INFO:teuthology.orchestra.run.smithi008.stdout: mon cluster log file level = debug 2024-04-26T01:06:22.888 INFO:teuthology.orchestra.run.smithi008.stdout: debug asserts on shutdown = true 2024-04-26T01:06:22.888 INFO:teuthology.orchestra.run.smithi008.stdout: mon health detail to clog = false 2024-04-26T01:06:22.888 INFO:teuthology.orchestra.run.smithi008.stdout: mon host = "172.21.15.8,172.21.15.12,172.21.15.116" 2024-04-26T01:06:22.889 INFO:teuthology.orchestra.run.smithi008.stdout: mon client directed command retry = 5 2024-04-26T01:06:22.889 INFO:teuthology.orchestra.run.smithi008.stdout: mon election default strategy = 1 2024-04-26T01:06:22.889 INFO:teuthology.orchestra.run.smithi008.stdout: ms inject delay max = 1 2024-04-26T01:06:22.889 INFO:teuthology.orchestra.run.smithi008.stdout: ms inject delay probability = 0.005 2024-04-26T01:06:22.889 INFO:teuthology.orchestra.run.smithi008.stdout: ms inject delay type = osd 2024-04-26T01:06:22.889 INFO:teuthology.orchestra.run.smithi008.stdout: ms inject internal delays = 0.002 2024-04-26T01:06:22.890 INFO:teuthology.orchestra.run.smithi008.stdout: ms inject socket failures = 2500 2024-04-26T01:06:22.890 INFO:teuthology.orchestra.run.smithi008.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-26T01:06:22.890 INFO:teuthology.orchestra.run.smithi008.stdout: fsid = 4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554 2024-04-26T01:06:22.890 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.890 INFO:teuthology.orchestra.run.smithi008.stdout:[osd] 2024-04-26T01:06:22.890 INFO:teuthology.orchestra.run.smithi008.stdout: osd journal size = 100 2024-04-26T01:06:22.891 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.891 INFO:teuthology.orchestra.run.smithi008.stdout: osd scrub load threshold = 5.0 2024-04-26T01:06:22.891 INFO:teuthology.orchestra.run.smithi008.stdout: osd scrub max interval = 120 2024-04-26T01:06:22.891 INFO:teuthology.orchestra.run.smithi008.stdout: osd mclock profile = high_recovery_ops 2024-04-26T01:06:22.891 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.891 INFO:teuthology.orchestra.run.smithi008.stdout: osd recover clone overlap = true 2024-04-26T01:06:22.891 INFO:teuthology.orchestra.run.smithi008.stdout: osd recovery max chunk = 1048576 2024-04-26T01:06:22.891 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.892 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug shutdown = true 2024-04-26T01:06:22.892 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug op order = true 2024-04-26T01:06:22.892 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug verify stray on activate = true 2024-04-26T01:06:22.892 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug trim objects = true 2024-04-26T01:06:22.892 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.892 INFO:teuthology.orchestra.run.smithi008.stdout: osd open classes on start = true 2024-04-26T01:06:22.892 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug pg log writeout = true 2024-04-26T01:06:22.892 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.893 INFO:teuthology.orchestra.run.smithi008.stdout: osd deep scrub update digest min age = 30 2024-04-26T01:06:22.893 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.893 INFO:teuthology.orchestra.run.smithi008.stdout: osd map max advance = 10 2024-04-26T01:06:22.893 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.893 INFO:teuthology.orchestra.run.smithi008.stdout: journal zero on create = true 2024-04-26T01:06:22.893 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.894 INFO:teuthology.orchestra.run.smithi008.stdout: filestore ondisk finisher threads = 3 2024-04-26T01:06:22.894 INFO:teuthology.orchestra.run.smithi008.stdout: filestore apply finisher threads = 3 2024-04-26T01:06:22.894 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.894 INFO:teuthology.orchestra.run.smithi008.stdout: bdev debug aio = true 2024-04-26T01:06:22.894 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug misdirected ops = true 2024-04-26T01:06:22.894 INFO:teuthology.orchestra.run.smithi008.stdout: debug monc = 20 2024-04-26T01:06:22.895 INFO:teuthology.orchestra.run.smithi008.stdout: bluestore block size = 96636764160 2024-04-26T01:06:22.895 INFO:teuthology.orchestra.run.smithi008.stdout: bluestore compression algorithm = lz4 2024-04-26T01:06:22.895 INFO:teuthology.orchestra.run.smithi008.stdout: bluestore compression mode = aggressive 2024-04-26T01:06:22.895 INFO:teuthology.orchestra.run.smithi008.stdout: bluestore fsck on mount = True 2024-04-26T01:06:22.895 INFO:teuthology.orchestra.run.smithi008.stdout: bluestore zero block detection = True 2024-04-26T01:06:22.895 INFO:teuthology.orchestra.run.smithi008.stdout: debug bluefs = 20 2024-04-26T01:06:22.896 INFO:teuthology.orchestra.run.smithi008.stdout: debug bluestore = 20 2024-04-26T01:06:22.896 INFO:teuthology.orchestra.run.smithi008.stdout: debug ms = 1 2024-04-26T01:06:22.896 INFO:teuthology.orchestra.run.smithi008.stdout: debug osd = 20 2024-04-26T01:06:22.896 INFO:teuthology.orchestra.run.smithi008.stdout: debug rocksdb = 10 2024-04-26T01:06:22.896 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T01:06:22.896 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd full ratio = 0.9 2024-04-26T01:06:22.896 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd nearfull ratio = 0.8 2024-04-26T01:06:22.897 INFO:teuthology.orchestra.run.smithi008.stdout: osd blocked scrub grace period = 3600 2024-04-26T01:06:22.897 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug reject backfill probability = 0.1 2024-04-26T01:06:22.897 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug verify cached snaps = True 2024-04-26T01:06:22.897 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug verify missing on start = True 2024-04-26T01:06:22.897 INFO:teuthology.orchestra.run.smithi008.stdout: osd failsafe full ratio = 0.95 2024-04-26T01:06:22.897 INFO:teuthology.orchestra.run.smithi008.stdout: osd max backfills = 3 2024-04-26T01:06:22.898 INFO:teuthology.orchestra.run.smithi008.stdout: osd max markdown count = 1000 2024-04-26T01:06:22.898 INFO:teuthology.orchestra.run.smithi008.stdout: osd mclock override recovery settings = True 2024-04-26T01:06:22.898 INFO:teuthology.orchestra.run.smithi008.stdout: osd objectstore = bluestore 2024-04-26T01:06:22.898 INFO:teuthology.orchestra.run.smithi008.stdout: osd op queue = debug_random 2024-04-26T01:06:22.898 INFO:teuthology.orchestra.run.smithi008.stdout: osd op queue cut off = debug_random 2024-04-26T01:06:22.898 INFO:teuthology.orchestra.run.smithi008.stdout: osd scrub min interval = 60 2024-04-26T01:06:22.899 INFO:teuthology.orchestra.run.smithi008.stdout: osd shutdown pgref assert = True 2024-04-26T01:06:22.899 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.899 INFO:teuthology.orchestra.run.smithi008.stdout:[mgr] 2024-04-26T01:06:22.899 INFO:teuthology.orchestra.run.smithi008.stdout: debug ms = 1 2024-04-26T01:06:22.899 INFO:teuthology.orchestra.run.smithi008.stdout: debug mgr = 20 2024-04-26T01:06:22.899 INFO:teuthology.orchestra.run.smithi008.stdout: debug mon = 20 2024-04-26T01:06:22.900 INFO:teuthology.orchestra.run.smithi008.stdout: debug auth = 20 2024-04-26T01:06:22.900 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:06:22.900 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:06:22.900 INFO:teuthology.orchestra.run.smithi008.stdout: mgr/telemetry/nag = false 2024-04-26T01:06:22.900 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.900 INFO:teuthology.orchestra.run.smithi008.stdout:[mon] 2024-04-26T01:06:22.900 INFO:teuthology.orchestra.run.smithi008.stdout: debug ms = 1 2024-04-26T01:06:22.901 INFO:teuthology.orchestra.run.smithi008.stdout: debug mon = 20 2024-04-26T01:06:22.901 INFO:teuthology.orchestra.run.smithi008.stdout: debug paxos = 20 2024-04-26T01:06:22.901 INFO:teuthology.orchestra.run.smithi008.stdout: debug auth = 20 2024-04-26T01:06:22.901 INFO:teuthology.orchestra.run.smithi008.stdout: mon data avail warn = 5 2024-04-26T01:06:22.901 INFO:teuthology.orchestra.run.smithi008.stdout: mon mgr mkfs grace = 240 2024-04-26T01:06:22.902 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:06:22.902 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd reporter subtree level = osd 2024-04-26T01:06:22.902 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd prime pg temp = true 2024-04-26T01:06:22.902 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:06:22.902 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.902 INFO:teuthology.orchestra.run.smithi008.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T01:06:22.902 INFO:teuthology.orchestra.run.smithi008.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T01:06:22.903 INFO:teuthology.orchestra.run.smithi008.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T01:06:22.903 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.903 INFO:teuthology.orchestra.run.smithi008.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T01:06:22.903 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T01:06:22.903 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T01:06:22.903 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.904 INFO:teuthology.orchestra.run.smithi008.stdout: # 1m isn't quite enough 2024-04-26T01:06:22.904 INFO:teuthology.orchestra.run.smithi008.stdout: mon_down_mkfs_grace = 2m 2024-04-26T01:06:22.904 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.904 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_filestore_osds = false 2024-04-26T01:06:22.904 INFO:teuthology.orchestra.run.smithi008.stdout: mon scrub interval = 300 2024-04-26T01:06:22.904 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:06:22.905 INFO:teuthology.orchestra.run.smithi008.stdout:[client] 2024-04-26T01:06:22.905 INFO:teuthology.orchestra.run.smithi008.stdout: rgw cache enabled = true 2024-04-26T01:06:22.905 INFO:teuthology.orchestra.run.smithi008.stdout: rgw enable ops log = true 2024-04-26T01:06:22.905 INFO:teuthology.orchestra.run.smithi008.stdout: rgw enable usage log = true 2024-04-26T01:06:22.905 INFO:teuthology.orchestra.run.smithi008.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T01:06:22.905 INFO:teuthology.orchestra.run.smithi008.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T01:06:22.905 INFO:teuthology.orchestra.run.smithi008.stdout:[mon.a] 2024-04-26T01:06:22.905 INFO:teuthology.orchestra.run.smithi008.stdout:[mon.b] 2024-04-26T01:06:22.905 INFO:teuthology.orchestra.run.smithi008.stdout:[mon.c] 2024-04-26T01:06:22.910 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2024-04-26T01:06:22.910 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2024-04-26T01:06:22.910 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T01:06:22.910 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2024-04-26T01:06:22.910 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.910 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2024-04-26T01:06:22.910 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.910 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2024-04-26T01:06:22.910 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.911 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2024-04-26T01:06:22.911 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2024-04-26T01:06:22.911 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.911 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2024-04-26T01:06:22.911 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2024-04-26T01:06:22.911 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.911 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T01:06:22.911 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2024-04-26T01:06:22.911 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.911 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T01:06:22.912 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T01:06:22.912 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.912 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2024-04-26T01:06:22.912 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.912 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2024-04-26T01:06:22.912 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2024-04-26T01:06:22.912 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2024-04-26T01:06:22.912 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2024-04-26T01:06:22.912 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2024-04-26T01:06:22.912 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2024-04-26T01:06:22.912 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2024-04-26T01:06:22.912 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T01:06:22.912 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T01:06:22.912 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2024-04-26T01:06:22.913 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.913 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T01:06:22.913 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.913 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2024-04-26T01:06:22.913 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.913 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2024-04-26T01:06:22.913 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.913 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2024-04-26T01:06:22.913 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2024-04-26T01:06:22.913 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2024-04-26T01:06:22.913 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = "172.21.15.8,172.21.15.12,172.21.15.116" 2024-04-26T01:06:22.913 INFO:teuthology.orchestra.run.smithi164.stdout: mon client directed command retry = 5 2024-04-26T01:06:22.913 INFO:teuthology.orchestra.run.smithi164.stdout: mon election default strategy = 1 2024-04-26T01:06:22.913 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject delay max = 1 2024-04-26T01:06:22.913 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject delay probability = 0.005 2024-04-26T01:06:22.914 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject delay type = osd 2024-04-26T01:06:22.914 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject internal delays = 0.002 2024-04-26T01:06:22.914 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject socket failures = 2500 2024-04-26T01:06:22.914 INFO:teuthology.orchestra.run.smithi164.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-26T01:06:22.914 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = 4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554 2024-04-26T01:06:22.914 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.914 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2024-04-26T01:06:22.914 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2024-04-26T01:06:22.914 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.914 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2024-04-26T01:06:22.914 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 120 2024-04-26T01:06:22.914 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2024-04-26T01:06:22.914 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.915 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2024-04-26T01:06:22.915 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2024-04-26T01:06:22.915 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.915 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2024-04-26T01:06:22.915 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2024-04-26T01:06:22.915 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2024-04-26T01:06:22.915 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2024-04-26T01:06:22.915 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.915 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2024-04-26T01:06:22.915 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2024-04-26T01:06:22.915 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.915 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2024-04-26T01:06:22.915 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.915 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2024-04-26T01:06:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2024-04-26T01:06:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2024-04-26T01:06:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2024-04-26T01:06:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2024-04-26T01:06:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2024-04-26T01:06:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: debug monc = 20 2024-04-26T01:06:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2024-04-26T01:06:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore compression algorithm = lz4 2024-04-26T01:06:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore compression mode = aggressive 2024-04-26T01:06:22.917 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2024-04-26T01:06:22.917 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore zero block detection = True 2024-04-26T01:06:22.917 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 20 2024-04-26T01:06:22.917 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 20 2024-04-26T01:06:22.917 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-04-26T01:06:22.917 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2024-04-26T01:06:22.917 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 10 2024-04-26T01:06:22.917 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T01:06:22.917 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2024-04-26T01:06:22.917 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2024-04-26T01:06:22.917 INFO:teuthology.orchestra.run.smithi164.stdout: osd blocked scrub grace period = 3600 2024-04-26T01:06:22.917 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug reject backfill probability = 0.1 2024-04-26T01:06:22.917 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify cached snaps = True 2024-04-26T01:06:22.918 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify missing on start = True 2024-04-26T01:06:22.918 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2024-04-26T01:06:22.918 INFO:teuthology.orchestra.run.smithi164.stdout: osd max backfills = 3 2024-04-26T01:06:22.918 INFO:teuthology.orchestra.run.smithi164.stdout: osd max markdown count = 1000 2024-04-26T01:06:22.918 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock override recovery settings = True 2024-04-26T01:06:22.918 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2024-04-26T01:06:22.918 INFO:teuthology.orchestra.run.smithi164.stdout: osd op queue = debug_random 2024-04-26T01:06:22.918 INFO:teuthology.orchestra.run.smithi164.stdout: osd op queue cut off = debug_random 2024-04-26T01:06:22.918 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub min interval = 60 2024-04-26T01:06:22.918 INFO:teuthology.orchestra.run.smithi164.stdout: osd shutdown pgref assert = True 2024-04-26T01:06:22.918 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.918 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2024-04-26T01:06:22.918 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-04-26T01:06:22.919 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2024-04-26T01:06:22.919 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-04-26T01:06:22.919 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-04-26T01:06:22.919 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:06:22.919 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:06:22.919 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2024-04-26T01:06:22.919 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.919 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2024-04-26T01:06:22.919 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-04-26T01:06:22.919 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-04-26T01:06:22.919 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2024-04-26T01:06:22.919 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-04-26T01:06:22.919 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2024-04-26T01:06:22.919 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2024-04-26T01:06:22.920 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:06:22.920 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2024-04-26T01:06:22.920 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2024-04-26T01:06:22.920 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:06:22.920 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.920 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T01:06:22.920 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T01:06:22.920 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T01:06:22.920 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.920 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T01:06:22.920 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T01:06:22.920 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T01:06:22.920 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.920 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2024-04-26T01:06:22.921 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2024-04-26T01:06:22.921 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.921 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2024-04-26T01:06:22.921 INFO:teuthology.orchestra.run.smithi164.stdout: mon scrub interval = 300 2024-04-26T01:06:22.921 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-26T01:06:22.921 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2024-04-26T01:06:22.921 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2024-04-26T01:06:22.921 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2024-04-26T01:06:22.921 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2024-04-26T01:06:22.921 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T01:06:22.921 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T01:06:22.921 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2024-04-26T01:06:22.921 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.b] 2024-04-26T01:06:22.921 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.c] 2024-04-26T01:06:22.922 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-04-26T01:06:22.922 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-04-26T01:06:22.922 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T01:06:22.922 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-04-26T01:06:22.922 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.922 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-04-26T01:06:22.922 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.922 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-04-26T01:06:22.922 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.922 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-04-26T01:06:22.922 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-04-26T01:06:22.922 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.922 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-04-26T01:06:22.923 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-04-26T01:06:22.923 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.923 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T01:06:22.923 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-04-26T01:06:22.923 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.923 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T01:06:22.923 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T01:06:22.923 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.923 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-04-26T01:06:22.923 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.923 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-04-26T01:06:22.923 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-04-26T01:06:22.923 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-04-26T01:06:22.924 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-04-26T01:06:22.924 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-04-26T01:06:22.924 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-04-26T01:06:22.924 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-04-26T01:06:22.924 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T01:06:22.924 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T01:06:22.924 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-04-26T01:06:22.924 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.924 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T01:06:22.924 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.924 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-04-26T01:06:22.924 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.924 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-04-26T01:06:22.924 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.924 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-04-26T01:06:22.925 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-04-26T01:06:22.925 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-04-26T01:06:22.925 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = "172.21.15.8,172.21.15.12,172.21.15.116" 2024-04-26T01:06:22.925 INFO:teuthology.orchestra.run.smithi116.stdout: mon client directed command retry = 5 2024-04-26T01:06:22.925 INFO:teuthology.orchestra.run.smithi116.stdout: mon election default strategy = 1 2024-04-26T01:06:22.925 INFO:teuthology.orchestra.run.smithi116.stdout: ms inject delay max = 1 2024-04-26T01:06:22.925 INFO:teuthology.orchestra.run.smithi116.stdout: ms inject delay probability = 0.005 2024-04-26T01:06:22.925 INFO:teuthology.orchestra.run.smithi116.stdout: ms inject delay type = osd 2024-04-26T01:06:22.925 INFO:teuthology.orchestra.run.smithi116.stdout: ms inject internal delays = 0.002 2024-04-26T01:06:22.925 INFO:teuthology.orchestra.run.smithi116.stdout: ms inject socket failures = 2500 2024-04-26T01:06:22.925 INFO:teuthology.orchestra.run.smithi116.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-26T01:06:22.925 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = 4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554 2024-04-26T01:06:22.925 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.925 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-04-26T01:06:22.926 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-04-26T01:06:22.926 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.926 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-04-26T01:06:22.926 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 120 2024-04-26T01:06:22.926 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-04-26T01:06:22.926 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.926 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-04-26T01:06:22.926 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-04-26T01:06:22.926 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.926 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-04-26T01:06:22.926 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-04-26T01:06:22.926 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-04-26T01:06:22.926 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug trim objects = true 2024-04-26T01:06:22.926 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.927 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-04-26T01:06:22.927 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-04-26T01:06:22.927 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.927 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-04-26T01:06:22.927 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.927 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-04-26T01:06:22.927 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.927 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-04-26T01:06:22.927 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.927 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-04-26T01:06:22.927 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-04-26T01:06:22.927 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.928 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-04-26T01:06:22.928 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-04-26T01:06:22.928 INFO:teuthology.orchestra.run.smithi116.stdout: debug monc = 20 2024-04-26T01:06:22.928 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2024-04-26T01:06:22.928 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore compression algorithm = lz4 2024-04-26T01:06:22.928 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore compression mode = aggressive 2024-04-26T01:06:22.928 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2024-04-26T01:06:22.928 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore zero block detection = True 2024-04-26T01:06:22.928 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 20 2024-04-26T01:06:22.928 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 20 2024-04-26T01:06:22.928 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-04-26T01:06:22.928 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-04-26T01:06:22.928 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 10 2024-04-26T01:06:22.928 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T01:06:22.928 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2024-04-26T01:06:22.929 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2024-04-26T01:06:22.929 INFO:teuthology.orchestra.run.smithi116.stdout: osd blocked scrub grace period = 3600 2024-04-26T01:06:22.929 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug reject backfill probability = 0.1 2024-04-26T01:06:22.929 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify cached snaps = True 2024-04-26T01:06:22.929 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify missing on start = True 2024-04-26T01:06:22.929 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2024-04-26T01:06:22.929 INFO:teuthology.orchestra.run.smithi116.stdout: osd max backfills = 3 2024-04-26T01:06:22.929 INFO:teuthology.orchestra.run.smithi116.stdout: osd max markdown count = 1000 2024-04-26T01:06:22.929 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock override recovery settings = True 2024-04-26T01:06:22.930 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2024-04-26T01:06:22.930 INFO:teuthology.orchestra.run.smithi116.stdout: osd op queue = debug_random 2024-04-26T01:06:22.930 INFO:teuthology.orchestra.run.smithi116.stdout: osd op queue cut off = debug_random 2024-04-26T01:06:22.930 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub min interval = 60 2024-04-26T01:06:22.930 INFO:teuthology.orchestra.run.smithi116.stdout: osd shutdown pgref assert = True 2024-04-26T01:06:22.930 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.930 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-04-26T01:06:22.931 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-04-26T01:06:22.931 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-04-26T01:06:22.931 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-04-26T01:06:22.931 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-04-26T01:06:22.931 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:06:22.931 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:06:22.931 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-04-26T01:06:22.931 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.932 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-04-26T01:06:22.932 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-04-26T01:06:22.932 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-04-26T01:06:22.932 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-04-26T01:06:22.932 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-04-26T01:06:22.932 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-04-26T01:06:22.933 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-04-26T01:06:22.933 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:06:22.933 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-04-26T01:06:22.933 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-04-26T01:06:22.933 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:06:22.933 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.934 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T01:06:22.934 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T01:06:22.934 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T01:06:22.934 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.934 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T01:06:22.934 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T01:06:22.934 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T01:06:22.935 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.935 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-04-26T01:06:22.935 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-04-26T01:06:22.935 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.935 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-04-26T01:06:22.935 INFO:teuthology.orchestra.run.smithi116.stdout: mon scrub interval = 300 2024-04-26T01:06:22.935 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-26T01:06:22.935 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-04-26T01:06:22.936 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-04-26T01:06:22.936 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-04-26T01:06:22.936 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-04-26T01:06:22.936 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T01:06:22.936 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T01:06:22.936 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-04-26T01:06:22.936 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.b] 2024-04-26T01:06:22.936 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.c] 2024-04-26T01:06:22.937 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2024-04-26T01:06:22.937 INFO:teuthology.orchestra.run.smithi012.stdout: chdir = "" 2024-04-26T01:06:22.937 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T01:06:22.937 INFO:teuthology.orchestra.run.smithi012.stdout: auth supported = cephx 2024-04-26T01:06:22.937 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.937 INFO:teuthology.orchestra.run.smithi012.stdout: filestore xattr use omap = true 2024-04-26T01:06:22.938 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.938 INFO:teuthology.orchestra.run.smithi012.stdout: mon clock drift allowed = 1.000 2024-04-26T01:06:22.938 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.938 INFO:teuthology.orchestra.run.smithi012.stdout: osd crush chooseleaf type = 0 2024-04-26T01:06:22.938 INFO:teuthology.orchestra.run.smithi012.stdout: auth debug = true 2024-04-26T01:06:22.939 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.939 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on old message = true 2024-04-26T01:06:22.939 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on bug = true 2024-04-26T01:06:22.939 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.939 INFO:teuthology.orchestra.run.smithi012.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T01:06:22.939 INFO:teuthology.orchestra.run.smithi012.stdout: mon pg warn max object skew = 0 2024-04-26T01:06:22.939 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.940 INFO:teuthology.orchestra.run.smithi012.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T01:06:22.940 INFO:teuthology.orchestra.run.smithi012.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T01:06:22.940 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.940 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default size = 2 2024-04-26T01:06:22.940 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.940 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow primary affinity = true 2024-04-26T01:06:22.940 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow pg remap = true 2024-04-26T01:06:22.940 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on legacy crush tunables = false 2024-04-26T01:06:22.940 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on crush straw calc version zero = false 2024-04-26T01:06:22.940 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on no sortbitwise = false 2024-04-26T01:06:22.940 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on osd down out interval zero = false 2024-04-26T01:06:22.940 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on too few osds = false 2024-04-26T01:06:22.941 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T01:06:22.941 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T01:06:22.941 INFO:teuthology.orchestra.run.smithi012.stdout: mon_allow_pool_size_one = true 2024-04-26T01:06:22.941 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.941 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T01:06:22.941 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.941 INFO:teuthology.orchestra.run.smithi012.stdout: osd default data pool replay window = 5 2024-04-26T01:06:22.941 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.941 INFO:teuthology.orchestra.run.smithi012.stdout: mon allow pool delete = true 2024-04-26T01:06:22.941 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.941 INFO:teuthology.orchestra.run.smithi012.stdout: mon cluster log file level = debug 2024-04-26T01:06:22.941 INFO:teuthology.orchestra.run.smithi012.stdout: debug asserts on shutdown = true 2024-04-26T01:06:22.941 INFO:teuthology.orchestra.run.smithi012.stdout: mon health detail to clog = false 2024-04-26T01:06:22.941 INFO:teuthology.orchestra.run.smithi012.stdout: mon host = "172.21.15.8,172.21.15.12,172.21.15.116" 2024-04-26T01:06:22.942 INFO:teuthology.orchestra.run.smithi012.stdout: mon client directed command retry = 5 2024-04-26T01:06:22.942 INFO:teuthology.orchestra.run.smithi012.stdout: mon election default strategy = 1 2024-04-26T01:06:22.942 INFO:teuthology.orchestra.run.smithi012.stdout: ms inject delay max = 1 2024-04-26T01:06:22.942 INFO:teuthology.orchestra.run.smithi012.stdout: ms inject delay probability = 0.005 2024-04-26T01:06:22.942 INFO:teuthology.orchestra.run.smithi012.stdout: ms inject delay type = osd 2024-04-26T01:06:22.942 INFO:teuthology.orchestra.run.smithi012.stdout: ms inject internal delays = 0.002 2024-04-26T01:06:22.942 INFO:teuthology.orchestra.run.smithi012.stdout: ms inject socket failures = 2500 2024-04-26T01:06:22.942 INFO:teuthology.orchestra.run.smithi012.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-26T01:06:22.942 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = 4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554 2024-04-26T01:06:22.942 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.942 INFO:teuthology.orchestra.run.smithi012.stdout:[osd] 2024-04-26T01:06:22.942 INFO:teuthology.orchestra.run.smithi012.stdout: osd journal size = 100 2024-04-26T01:06:22.942 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.943 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub load threshold = 5.0 2024-04-26T01:06:22.943 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub max interval = 120 2024-04-26T01:06:22.943 INFO:teuthology.orchestra.run.smithi012.stdout: osd mclock profile = high_recovery_ops 2024-04-26T01:06:22.943 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.943 INFO:teuthology.orchestra.run.smithi012.stdout: osd recover clone overlap = true 2024-04-26T01:06:22.943 INFO:teuthology.orchestra.run.smithi012.stdout: osd recovery max chunk = 1048576 2024-04-26T01:06:22.943 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.943 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug shutdown = true 2024-04-26T01:06:22.943 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug op order = true 2024-04-26T01:06:22.943 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify stray on activate = true 2024-04-26T01:06:22.943 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug trim objects = true 2024-04-26T01:06:22.943 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.944 INFO:teuthology.orchestra.run.smithi012.stdout: osd open classes on start = true 2024-04-26T01:06:22.944 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug pg log writeout = true 2024-04-26T01:06:22.944 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.944 INFO:teuthology.orchestra.run.smithi012.stdout: osd deep scrub update digest min age = 30 2024-04-26T01:06:22.944 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.944 INFO:teuthology.orchestra.run.smithi012.stdout: osd map max advance = 10 2024-04-26T01:06:22.944 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.944 INFO:teuthology.orchestra.run.smithi012.stdout: journal zero on create = true 2024-04-26T01:06:22.944 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.944 INFO:teuthology.orchestra.run.smithi012.stdout: filestore ondisk finisher threads = 3 2024-04-26T01:06:22.944 INFO:teuthology.orchestra.run.smithi012.stdout: filestore apply finisher threads = 3 2024-04-26T01:06:22.945 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.945 INFO:teuthology.orchestra.run.smithi012.stdout: bdev debug aio = true 2024-04-26T01:06:22.945 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug misdirected ops = true 2024-04-26T01:06:22.945 INFO:teuthology.orchestra.run.smithi012.stdout: debug monc = 20 2024-04-26T01:06:22.945 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore block size = 96636764160 2024-04-26T01:06:22.945 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore compression algorithm = lz4 2024-04-26T01:06:22.945 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore compression mode = aggressive 2024-04-26T01:06:22.945 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore fsck on mount = True 2024-04-26T01:06:22.945 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore zero block detection = True 2024-04-26T01:06:22.945 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluefs = 20 2024-04-26T01:06:22.945 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluestore = 20 2024-04-26T01:06:22.945 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-04-26T01:06:22.945 INFO:teuthology.orchestra.run.smithi012.stdout: debug osd = 20 2024-04-26T01:06:22.946 INFO:teuthology.orchestra.run.smithi012.stdout: debug rocksdb = 10 2024-04-26T01:06:22.946 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T01:06:22.946 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd full ratio = 0.9 2024-04-26T01:06:22.946 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd nearfull ratio = 0.8 2024-04-26T01:06:22.946 INFO:teuthology.orchestra.run.smithi012.stdout: osd blocked scrub grace period = 3600 2024-04-26T01:06:22.946 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug reject backfill probability = 0.1 2024-04-26T01:06:22.946 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify cached snaps = True 2024-04-26T01:06:22.946 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify missing on start = True 2024-04-26T01:06:22.946 INFO:teuthology.orchestra.run.smithi012.stdout: osd failsafe full ratio = 0.95 2024-04-26T01:06:22.946 INFO:teuthology.orchestra.run.smithi012.stdout: osd max backfills = 3 2024-04-26T01:06:22.946 INFO:teuthology.orchestra.run.smithi012.stdout: osd max markdown count = 1000 2024-04-26T01:06:22.946 INFO:teuthology.orchestra.run.smithi012.stdout: osd mclock override recovery settings = True 2024-04-26T01:06:22.946 INFO:teuthology.orchestra.run.smithi012.stdout: osd objectstore = bluestore 2024-04-26T01:06:22.947 INFO:teuthology.orchestra.run.smithi012.stdout: osd op queue = debug_random 2024-04-26T01:06:22.947 INFO:teuthology.orchestra.run.smithi012.stdout: osd op queue cut off = debug_random 2024-04-26T01:06:22.947 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub min interval = 60 2024-04-26T01:06:22.947 INFO:teuthology.orchestra.run.smithi012.stdout: osd shutdown pgref assert = True 2024-04-26T01:06:22.947 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.947 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr] 2024-04-26T01:06:22.947 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-04-26T01:06:22.947 INFO:teuthology.orchestra.run.smithi012.stdout: debug mgr = 20 2024-04-26T01:06:22.947 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2024-04-26T01:06:22.947 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2024-04-26T01:06:22.947 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:06:22.947 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:06:22.947 INFO:teuthology.orchestra.run.smithi012.stdout: mgr/telemetry/nag = false 2024-04-26T01:06:22.947 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.947 INFO:teuthology.orchestra.run.smithi012.stdout:[mon] 2024-04-26T01:06:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-04-26T01:06:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2024-04-26T01:06:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: debug paxos = 20 2024-04-26T01:06:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2024-04-26T01:06:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: mon data avail warn = 5 2024-04-26T01:06:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: mon mgr mkfs grace = 240 2024-04-26T01:06:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:06:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd reporter subtree level = osd 2024-04-26T01:06:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd prime pg temp = true 2024-04-26T01:06:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:06:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T01:06:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T01:06:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T01:06:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T01:06:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T01:06:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T01:06:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: # 1m isn't quite enough 2024-04-26T01:06:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: mon_down_mkfs_grace = 2m 2024-04-26T01:06:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_filestore_osds = false 2024-04-26T01:06:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: mon scrub interval = 300 2024-04-26T01:06:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-26T01:06:22.949 INFO:teuthology.orchestra.run.smithi012.stdout:[client] 2024-04-26T01:06:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: rgw cache enabled = true 2024-04-26T01:06:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable ops log = true 2024-04-26T01:06:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable usage log = true 2024-04-26T01:06:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T01:06:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T01:06:22.950 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.a] 2024-04-26T01:06:22.950 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.b] 2024-04-26T01:06:22.950 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.c] 2024-04-26T01:06:22.950 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-26T01:06:22.950 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-26T01:06:23.001 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-26T01:06:23.001 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:06:23.001 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-26T01:06:23.007 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:06:23.007 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-26T01:06:23.056 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi008.front.sepia.ceph.com 2024-04-26T01:06:23.056 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:06:23.056 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T01:06:23.057 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T01:06:23.122 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:06:23.122 DEBUG:teuthology.orchestra.run.smithi008:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:23.168 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi012.front.sepia.ceph.com 2024-04-26T01:06:23.168 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:06:23.168 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T01:06:23.168 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T01:06:23.193 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:06:23.193 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:23.240 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi116.front.sepia.ceph.com 2024-04-26T01:06:23.240 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:06:23.240 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T01:06:23.240 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T01:06:23.262 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:06:23.263 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:23.311 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi164.front.sepia.ceph.com 2024-04-26T01:06:23.311 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:06:23.311 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T01:06:23.311 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T01:06:23.335 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:06:23.335 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:23.383 INFO:tasks.ceph:Setting up mon nodes... 2024-04-26T01:06:23.383 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-26T01:06:23.384 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-26T01:06:23.428 INFO:teuthology.orchestra.run.smithi008.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-26T01:06:23.432 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T01:06:23.474 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T01:06:23.492 INFO:tasks.ceph:Setting up mds nodes... 2024-04-26T01:06:23.492 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-26T01:06:23.493 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T01:06:23.568 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-26T01:06:23.578 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-26T01:06:23.579 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi008.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-04-26T01:06:23.579 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-26T01:06:23.593 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:23.593 INFO:tasks.ceph:role: osd.0 2024-04-26T01:06:23.593 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi008.front.sepia.ceph.com 2024-04-26T01:06:23.593 DEBUG:teuthology.orchestra.run.smithi008:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T01:06:23.649 INFO:teuthology.orchestra.run.smithi008.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:06:23.649 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:06:23.649 INFO:teuthology.orchestra.run.smithi008.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:06:23.649 INFO:teuthology.orchestra.run.smithi008.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:06:23.649 INFO:teuthology.orchestra.run.smithi008.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:06:23.649 INFO:teuthology.orchestra.run.smithi008.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:06:23.649 INFO:teuthology.orchestra.run.smithi008.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:06:23.650 INFO:teuthology.orchestra.run.smithi008.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:06:23.650 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:06:23.650 INFO:teuthology.orchestra.run.smithi008.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:06:23.672 INFO:teuthology.orchestra.run.smithi008.stdout:Discarding blocks...Done. 2024-04-26T01:06:23.674 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi008.front.sepia.ceph.com -o noatime 2024-04-26T01:06:23.674 DEBUG:teuthology.orchestra.run.smithi008:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-26T01:06:23.730 DEBUG:teuthology.orchestra.run.smithi008:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-26T01:06:23.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:06:23.782 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:06:23.782 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-26T01:06:23.835 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:23.835 INFO:tasks.ceph:role: osd.4 2024-04-26T01:06:23.835 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi008.front.sepia.ceph.com 2024-04-26T01:06:23.836 DEBUG:teuthology.orchestra.run.smithi008:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T01:06:23.891 INFO:teuthology.orchestra.run.smithi008.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:06:23.891 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:06:23.892 INFO:teuthology.orchestra.run.smithi008.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:06:23.892 INFO:teuthology.orchestra.run.smithi008.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:06:23.892 INFO:teuthology.orchestra.run.smithi008.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:06:23.892 INFO:teuthology.orchestra.run.smithi008.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:06:23.892 INFO:teuthology.orchestra.run.smithi008.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:06:23.892 INFO:teuthology.orchestra.run.smithi008.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:06:23.892 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:06:23.892 INFO:teuthology.orchestra.run.smithi008.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:06:23.915 INFO:teuthology.orchestra.run.smithi008.stdout:Discarding blocks...Done. 2024-04-26T01:06:23.917 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi008.front.sepia.ceph.com -o noatime 2024-04-26T01:06:23.917 DEBUG:teuthology.orchestra.run.smithi008:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-04-26T01:06:23.981 DEBUG:teuthology.orchestra.run.smithi008:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-26T01:06:24.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:06:24.032 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:06:24.033 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-04-26T01:06:24.085 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:24.085 INFO:tasks.ceph:role: osd.8 2024-04-26T01:06:24.085 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi008.front.sepia.ceph.com 2024-04-26T01:06:24.085 DEBUG:teuthology.orchestra.run.smithi008:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T01:06:24.138 INFO:teuthology.orchestra.run.smithi008.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:06:24.138 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:06:24.138 INFO:teuthology.orchestra.run.smithi008.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:06:24.139 INFO:teuthology.orchestra.run.smithi008.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:06:24.139 INFO:teuthology.orchestra.run.smithi008.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:06:24.139 INFO:teuthology.orchestra.run.smithi008.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:06:24.139 INFO:teuthology.orchestra.run.smithi008.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:06:24.139 INFO:teuthology.orchestra.run.smithi008.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:06:24.139 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:06:24.139 INFO:teuthology.orchestra.run.smithi008.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:06:24.162 INFO:teuthology.orchestra.run.smithi008.stdout:Discarding blocks...Done. 2024-04-26T01:06:24.164 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi008.front.sepia.ceph.com -o noatime 2024-04-26T01:06:24.164 DEBUG:teuthology.orchestra.run.smithi008:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-04-26T01:06:24.225 DEBUG:teuthology.orchestra.run.smithi008:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-04-26T01:06:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:06:24.280 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:06:24.280 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-04-26T01:06:24.334 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:24.334 INFO:tasks.ceph:role: osd.12 2024-04-26T01:06:24.335 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi008.front.sepia.ceph.com 2024-04-26T01:06:24.335 DEBUG:teuthology.orchestra.run.smithi008:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-26T01:06:24.387 INFO:teuthology.orchestra.run.smithi008.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:06:24.388 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:06:24.388 INFO:teuthology.orchestra.run.smithi008.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:06:24.388 INFO:teuthology.orchestra.run.smithi008.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:06:24.388 INFO:teuthology.orchestra.run.smithi008.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:06:24.388 INFO:teuthology.orchestra.run.smithi008.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:06:24.388 INFO:teuthology.orchestra.run.smithi008.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:06:24.388 INFO:teuthology.orchestra.run.smithi008.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:06:24.388 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:06:24.388 INFO:teuthology.orchestra.run.smithi008.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:06:24.412 INFO:teuthology.orchestra.run.smithi008.stdout:Discarding blocks...Done. 2024-04-26T01:06:24.414 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi008.front.sepia.ceph.com -o noatime 2024-04-26T01:06:24.414 DEBUG:teuthology.orchestra.run.smithi008:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-04-26T01:06:24.472 DEBUG:teuthology.orchestra.run.smithi008:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-04-26T01:06:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:06:24.524 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:06:24.524 DEBUG:teuthology.orchestra.run.smithi008:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:24.643 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:24.637+0000 7fe4b85ea7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-26T01:06:24.643 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:24.637+0000 7fe4b85ea7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-26T01:06:24.643 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:24.637+0000 7fe4b85ea7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T01:06:24.643 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:24.637+0000 7fe4b85ea7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T01:06:24.643 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:24.637+0000 7fe4b85ea7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T01:06:24.644 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:24.641+0000 7fe4b85ea7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-26T01:06:27.620 DEBUG:teuthology.orchestra.run.smithi008:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-26T01:06:27.635 DEBUG:teuthology.orchestra.run.smithi008:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:27.719 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:27.713+0000 7f3a987eb7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-26T01:06:27.719 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:27.713+0000 7f3a987eb7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-26T01:06:27.719 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:27.713+0000 7f3a987eb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T01:06:27.719 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:27.713+0000 7f3a987eb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T01:06:27.719 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:27.713+0000 7f3a987eb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T01:06:27.719 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:27.713+0000 7f3a987eb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-26T01:06:30.690 DEBUG:teuthology.orchestra.run.smithi008:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-26T01:06:30.743 DEBUG:teuthology.orchestra.run.smithi008:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:30.832 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:30.829+0000 7f45ed9117c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-04-26T01:06:30.832 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:30.829+0000 7f45ed9117c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-04-26T01:06:30.832 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:30.829+0000 7f45ed9117c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-26T01:06:30.832 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:30.829+0000 7f45ed9117c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-26T01:06:30.832 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:30.829+0000 7f45ed9117c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-26T01:06:30.832 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:30.829+0000 7f45ed9117c0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-04-26T01:06:33.826 DEBUG:teuthology.orchestra.run.smithi008:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-04-26T01:06:33.881 DEBUG:teuthology.orchestra.run.smithi008:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:33.965 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:33.961+0000 7f91113507c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-04-26T01:06:33.965 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:33.961+0000 7f91113507c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-04-26T01:06:33.965 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:33.961+0000 7f91113507c0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-04-26T01:06:33.965 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:33.961+0000 7f91113507c0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-04-26T01:06:33.965 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:33.961+0000 7f91113507c0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-04-26T01:06:33.965 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:06:33.961+0000 7f91113507c0 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-04-26T01:06:36.938 DEBUG:teuthology.orchestra.run.smithi008:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-04-26T01:06:36.992 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-26T01:06:37.007 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:37.007 INFO:tasks.ceph:role: osd.1 2024-04-26T01:06:37.007 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-26T01:06:37.008 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T01:06:37.073 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:06:37.073 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:06:37.073 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:06:37.073 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:06:37.073 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:06:37.073 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:06:37.074 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:06:37.074 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:06:37.074 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:06:37.074 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:06:37.518 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-04-26T01:06:37.521 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-04-26T01:06:37.521 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-04-26T01:06:37.616 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-26T01:06:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:06:37.626 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:06:37.626 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-26T01:06:37.683 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:37.683 INFO:tasks.ceph:role: osd.5 2024-04-26T01:06:37.683 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-26T01:06:37.683 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T01:06:37.748 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:06:37.748 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:06:37.748 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:06:37.748 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:06:37.748 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:06:37.748 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:06:37.748 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:06:37.749 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:06:37.749 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:06:37.749 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:06:38.212 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-04-26T01:06:38.214 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-04-26T01:06:38.214 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-26T01:06:38.362 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-26T01:06:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:06:38.375 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:06:38.375 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-04-26T01:06:38.431 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:38.431 INFO:tasks.ceph:role: osd.9 2024-04-26T01:06:38.432 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-26T01:06:38.432 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T01:06:38.496 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:06:38.496 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:06:38.496 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:06:38.496 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:06:38.496 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:06:38.496 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:06:38.496 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:06:38.496 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:06:38.496 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:06:38.496 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:06:38.973 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-04-26T01:06:38.975 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-04-26T01:06:38.975 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-04-26T01:06:39.122 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-04-26T01:06:39.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:06:39.135 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:06:39.135 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-04-26T01:06:39.191 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:39.191 INFO:tasks.ceph:role: osd.13 2024-04-26T01:06:39.191 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-26T01:06:39.191 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-26T01:06:39.254 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:06:39.254 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:06:39.255 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:06:39.255 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:06:39.255 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:06:39.255 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:06:39.255 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:06:39.255 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:06:39.255 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:06:39.255 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:06:39.690 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-04-26T01:06:39.692 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-04-26T01:06:39.692 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-04-26T01:06:39.780 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-04-26T01:06:39.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:06:39.789 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:06:39.790 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:39.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:39.900+0000 7fd701a1d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-26T01:06:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:39.900+0000 7fd701a1d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-26T01:06:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:39.900+0000 7fd701a1d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T01:06:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:39.900+0000 7fd701a1d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T01:06:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:39.900+0000 7fd701a1d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T01:06:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:39.908+0000 7fd701a1d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-26T01:06:42.920 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-26T01:06:42.935 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:43.012+0000 7fe646ed17c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-26T01:06:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:43.016+0000 7fe646ed17c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-26T01:06:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:43.016+0000 7fe646ed17c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T01:06:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:43.016+0000 7fe646ed17c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T01:06:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:43.016+0000 7fe646ed17c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T01:06:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:43.016+0000 7fe646ed17c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-26T01:06:45.995 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-26T01:06:46.056 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:46.136+0000 7f4ad4b2f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-04-26T01:06:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:46.136+0000 7f4ad4b2f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-04-26T01:06:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:46.136+0000 7f4ad4b2f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-26T01:06:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:46.136+0000 7f4ad4b2f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-26T01:06:46.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:46.136+0000 7f4ad4b2f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-26T01:06:46.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:46.136+0000 7f4ad4b2f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-04-26T01:06:49.109 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-04-26T01:06:49.164 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:49.244+0000 7fc32ffc27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-04-26T01:06:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:49.244+0000 7fc32ffc27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-04-26T01:06:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:49.244+0000 7fc32ffc27c0 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-04-26T01:06:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:49.244+0000 7fc32ffc27c0 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-04-26T01:06:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:49.244+0000 7fc32ffc27c0 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-04-26T01:06:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-26T01:06:49.244+0000 7fc32ffc27c0 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-04-26T01:06:52.227 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-04-26T01:06:52.284 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-26T01:06:52.298 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:52.298 INFO:tasks.ceph:role: osd.2 2024-04-26T01:06:52.299 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-26T01:06:52.299 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T01:06:52.361 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:06:52.362 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:06:52.362 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:06:52.362 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:06:52.362 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:06:52.362 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:06:52.362 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:06:52.362 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:06:52.362 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:06:52.363 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:06:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-04-26T01:06:52.849 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-04-26T01:06:52.849 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-04-26T01:06:52.994 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-26T01:06:53.005 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:06:53.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:06:53.006 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-26T01:06:53.061 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:53.061 INFO:tasks.ceph:role: osd.6 2024-04-26T01:06:53.062 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-26T01:06:53.062 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T01:06:53.126 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:06:53.126 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:06:53.126 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:06:53.126 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:06:53.126 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:06:53.126 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:06:53.127 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:06:53.127 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:06:53.127 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:06:53.127 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:06:53.570 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-04-26T01:06:53.572 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-04-26T01:06:53.572 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-04-26T01:06:53.661 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-26T01:06:53.672 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:06:53.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:06:53.673 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-04-26T01:06:53.729 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:53.730 INFO:tasks.ceph:role: osd.10 2024-04-26T01:06:53.730 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-26T01:06:53.730 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T01:06:53.793 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:06:53.793 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:06:53.793 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:06:53.793 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:06:53.794 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:06:53.794 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:06:53.794 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:06:53.794 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:06:53.794 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:06:53.794 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:06:54.252 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-04-26T01:06:54.254 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-04-26T01:06:54.254 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-04-26T01:06:54.363 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-04-26T01:06:54.373 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:06:54.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:06:54.375 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-04-26T01:06:54.430 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-04-26T01:06:54.430 INFO:tasks.ceph:role: osd.14 2024-04-26T01:06:54.431 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-26T01:06:54.431 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-26T01:06:54.494 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:06:54.494 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:06:54.494 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:06:54.494 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:06:54.494 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:06:54.494 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:06:54.494 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:06:54.494 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:06:54.494 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:06:54.494 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:06:54.981 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-04-26T01:06:54.983 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-04-26T01:06:54.984 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-04-26T01:06:55.120 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-04-26T01:06:55.131 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:06:55.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:06:55.132 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:06:55.248+0000 7fdb995d57c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-26T01:06:55.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:06:55.256+0000 7fdb995d57c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-26T01:06:55.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:06:55.256+0000 7fdb995d57c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T01:06:55.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:06:55.256+0000 7fdb995d57c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T01:06:55.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:06:55.256+0000 7fdb995d57c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T01:06:55.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:06:55.264+0000 7fdb995d57c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-26T01:06:58.245 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-26T01:06:58.259 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:06:58.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:06:58.340+0000 7fd397d027c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-26T01:06:58.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:06:58.340+0000 7fd397d027c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-26T01:06:58.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:06:58.340+0000 7fd397d027c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-26T01:06:58.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:06:58.340+0000 7fd397d027c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-26T01:06:58.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:06:58.340+0000 7fd397d027c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-26T01:06:58.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:06:58.340+0000 7fd397d027c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-26T01:07:01.309 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-26T01:07:01.370 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:07:01.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:07:01.448+0000 7fce6c5c87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-04-26T01:07:01.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:07:01.448+0000 7fce6c5c87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-04-26T01:07:01.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:07:01.448+0000 7fce6c5c87c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-04-26T01:07:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:07:01.448+0000 7fce6c5c87c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-04-26T01:07:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:07:01.448+0000 7fce6c5c87c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-04-26T01:07:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:07:01.448+0000 7fce6c5c87c0 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-04-26T01:07:04.433 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-04-26T01:07:04.488 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:07:04.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:07:04.568+0000 7fccf44f27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-04-26T01:07:04.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:07:04.568+0000 7fccf44f27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-04-26T01:07:04.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:07:04.568+0000 7fccf44f27c0 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-04-26T01:07:04.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:07:04.568+0000 7fccf44f27c0 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-04-26T01:07:04.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:07:04.568+0000 7fccf44f27c0 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-04-26T01:07:04.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-26T01:07:04.568+0000 7fccf44f27c0 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-04-26T01:07:07.533 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-04-26T01:07:07.593 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-26T01:07:07.607 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-04-26T01:07:07.608 INFO:tasks.ceph:role: osd.3 2024-04-26T01:07:07.608 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com 2024-04-26T01:07:07.608 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T01:07:07.671 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:07:07.671 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:07:07.671 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:07:07.671 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:07:07.671 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:07:07.672 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:07:07.672 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:07:07.672 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:07:07.672 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:07:07.672 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:07:08.163 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-04-26T01:07:08.166 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-04-26T01:07:08.166 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-04-26T01:07:08.303 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-26T01:07:08.314 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:07:08.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:07:08.315 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-26T01:07:08.371 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-04-26T01:07:08.371 INFO:tasks.ceph:role: osd.7 2024-04-26T01:07:08.371 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com 2024-04-26T01:07:08.371 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T01:07:08.436 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:07:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:07:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:07:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:07:08.436 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:07:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:07:08.436 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:07:08.436 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:07:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:07:08.436 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:07:08.875 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-04-26T01:07:08.877 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-04-26T01:07:08.877 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-04-26T01:07:08.938 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-26T01:07:08.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:07:08.991 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:07:08.991 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-04-26T01:07:09.044 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-04-26T01:07:09.044 INFO:tasks.ceph:role: osd.11 2024-04-26T01:07:09.044 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com 2024-04-26T01:07:09.044 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T01:07:09.106 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:07:09.107 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:07:09.107 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:07:09.107 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:07:09.107 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:07:09.107 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:07:09.107 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:07:09.107 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:07:09.107 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:07:09.107 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:07:09.569 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-04-26T01:07:09.571 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-04-26T01:07:09.571 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-04-26T01:07:09.667 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-04-26T01:07:09.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:07:09.679 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:07:09.679 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-04-26T01:07:09.735 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-04-26T01:07:09.735 INFO:tasks.ceph:role: osd.15 2024-04-26T01:07:09.735 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi164.front.sepia.ceph.com 2024-04-26T01:07:09.735 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-26T01:07:09.798 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:07:09.798 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:07:09.798 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:07:09.798 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:07:09.798 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:07:09.798 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:07:09.798 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:07:09.799 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:07:09.799 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:07:09.799 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:07:10.266 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-04-26T01:07:10.310 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-04-26T01:07:10.311 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-04-26T01:07:10.453 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-04-26T01:07:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:07:10.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:07:10.465 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:07:10.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:10.657+0000 7f4c39dcd7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-26T01:07:10.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:10.661+0000 7f4c39dcd7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-26T01:07:10.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:10.661+0000 7f4c39dcd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T01:07:10.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:10.661+0000 7f4c39dcd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T01:07:10.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:10.661+0000 7f4c39dcd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T01:07:10.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:10.669+0000 7f4c39dcd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-26T01:07:13.662 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-26T01:07:13.676 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:07:13.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:13.753+0000 7f3f2ffe87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-26T01:07:13.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:13.753+0000 7f3f2ffe87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-26T01:07:13.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:13.753+0000 7f3f2ffe87c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-26T01:07:13.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:13.753+0000 7f3f2ffe87c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-26T01:07:13.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:13.753+0000 7f3f2ffe87c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-26T01:07:13.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:13.757+0000 7f3f2ffe87c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-26T01:07:16.730 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-26T01:07:16.787 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:07:16.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:16.865+0000 7f2f04d927c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-04-26T01:07:16.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:16.865+0000 7f2f04d927c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-04-26T01:07:16.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:16.865+0000 7f2f04d927c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-04-26T01:07:16.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:16.865+0000 7f2f04d927c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-04-26T01:07:16.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:16.865+0000 7f2f04d927c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-04-26T01:07:16.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:16.869+0000 7f2f04d927c0 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-04-26T01:07:19.836 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-04-26T01:07:19.891 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:07:19.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:19.969+0000 7f2b7a89c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-04-26T01:07:19.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:19.969+0000 7f2b7a89c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-04-26T01:07:19.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:19.969+0000 7f2b7a89c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-04-26T01:07:19.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:19.969+0000 7f2b7a89c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-04-26T01:07:19.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:19.969+0000 7f2b7a89c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-04-26T01:07:19.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-26T01:07:19.969+0000 7f2b7a89c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-04-26T01:07:22.946 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-04-26T01:07:23.003 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-26T01:07:23.003 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:07:23.003 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-26T01:07:23.018 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:07:23.019 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-26T01:07:23.075 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:07:23.075 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-26T01:07:23.130 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:07:23.130 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-04-26T01:07:23.187 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:07:23.187 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-04-26T01:07:23.243 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:07:23.243 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-26T01:07:23.262 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:07:23.262 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-26T01:07:23.319 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:07:23.320 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-04-26T01:07:23.375 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:07:23.375 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-04-26T01:07:23.430 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:07:23.430 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-26T01:07:23.445 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:07:23.445 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-26T01:07:23.500 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:07:23.500 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-04-26T01:07:23.557 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:07:23.557 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-04-26T01:07:23.612 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:07:23.612 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-26T01:07:23.625 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:07:23.625 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-26T01:07:23.682 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:07:23.682 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-26T01:07:23.737 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:07:23.737 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-04-26T01:07:23.789 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:07:23.789 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-04-26T01:07:23.841 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:07:23.841 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-26T01:07:23.886 INFO:tasks.ceph:Adding keys to all mons... 2024-04-26T01:07:23.887 DEBUG:teuthology.orchestra.run.smithi008:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T01:07:23.890 DEBUG:teuthology.orchestra.run.smithi012:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T01:07:23.892 DEBUG:teuthology.orchestra.run.smithi116:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T01:07:23.899 INFO:teuthology.orchestra.run.smithi008.stdout:[mgr.y] 2024-04-26T01:07:23.899 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAP/ipmF618GRAAzhrROMUTWs51cvxMyRU5jw== 2024-04-26T01:07:23.899 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.0] 2024-04-26T01:07:23.899 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAQ/ipmaY87JhAAFRHHgcfZSG555xjSSJQDTQ== 2024-04-26T01:07:23.899 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.4] 2024-04-26T01:07:23.900 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAT/ipmQ/m+KhAAGaq0s4Hh7LbMALmtlqet3Q== 2024-04-26T01:07:23.900 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.8] 2024-04-26T01:07:23.900 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAW/ipmIh5/MRAAMqDYtjGww0fOjieYaL5iOQ== 2024-04-26T01:07:23.900 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.12] 2024-04-26T01:07:23.900 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAZ/ipmhSFuORAApFFyFZuxqyILqpfis2MQPA== 2024-04-26T01:07:23.900 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.1] 2024-04-26T01:07:23.900 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAf/ipmVxXeNRAABzQ2bhrQM94nTDSQCJMmOg== 2024-04-26T01:07:23.900 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.5] 2024-04-26T01:07:23.900 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAj/ipmzgD+ABAAJgIrgxJDPCKVRZlPLgLKow== 2024-04-26T01:07:23.900 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.9] 2024-04-26T01:07:23.900 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAm/ipmrzk0CBAA+xbOyJAwcvMl4yZdsr+LTg== 2024-04-26T01:07:23.900 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.13] 2024-04-26T01:07:23.900 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAp/ipmD3CuDhAAGKLWUtWdZC0yZ85NVFHTTA== 2024-04-26T01:07:23.901 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.2] 2024-04-26T01:07:23.901 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAv/ipmDgUEDxAAi/EW7aDj81VCuuOMhMvqYQ== 2024-04-26T01:07:23.901 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.6] 2024-04-26T01:07:23.901 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAy/ipmTUp9FBAANkjvGJlyN2gEVH+lwi5U3Q== 2024-04-26T01:07:23.901 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.10] 2024-04-26T01:07:23.901 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQA1/ipmV77gGhAAG8xnsr3DlF8dyvMdxDgJrg== 2024-04-26T01:07:23.901 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.14] 2024-04-26T01:07:23.901 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQA4/ipmoFEAIhAA0m06wDSzUwpXear3fBIQcw== 2024-04-26T01:07:23.901 INFO:teuthology.orchestra.run.smithi008.stdout:[mgr.x] 2024-04-26T01:07:23.901 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAP/ipms+dhHBAA8qvdsjdi2bG0e5qjHIBkNg== 2024-04-26T01:07:23.901 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.3] 2024-04-26T01:07:23.901 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQA+/ipmzs5zJxAAmMSRVGcv8OMdpB5E5amXrw== 2024-04-26T01:07:23.901 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.7] 2024-04-26T01:07:23.901 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQBB/ipmNiVJLRAAb7mQrRcGo9VJdCvTozWpug== 2024-04-26T01:07:23.902 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.11] 2024-04-26T01:07:23.902 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQBE/ipmZ/H2MxAAFDP+GcrxtiLgE38sspRfmg== 2024-04-26T01:07:23.902 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.15] 2024-04-26T01:07:23.902 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQBH/ipmd1UdOhAAJ9rhb7VNVFKKvF4m7q3gBw== 2024-04-26T01:07:23.902 INFO:teuthology.orchestra.run.smithi008.stdout:[client.0] 2024-04-26T01:07:23.902 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAP/ipmINz3IRAA4iN718g6n9G90IaxjaT2mg== 2024-04-26T01:07:23.902 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr.y] 2024-04-26T01:07:23.903 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAP/ipmF618GRAAzhrROMUTWs51cvxMyRU5jw== 2024-04-26T01:07:23.903 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.0] 2024-04-26T01:07:23.903 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAQ/ipmaY87JhAAFRHHgcfZSG555xjSSJQDTQ== 2024-04-26T01:07:23.903 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.4] 2024-04-26T01:07:23.903 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAT/ipmQ/m+KhAAGaq0s4Hh7LbMALmtlqet3Q== 2024-04-26T01:07:23.903 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.8] 2024-04-26T01:07:23.903 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAW/ipmIh5/MRAAMqDYtjGww0fOjieYaL5iOQ== 2024-04-26T01:07:23.903 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.12] 2024-04-26T01:07:23.903 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAZ/ipmhSFuORAApFFyFZuxqyILqpfis2MQPA== 2024-04-26T01:07:23.903 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.1] 2024-04-26T01:07:23.903 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAf/ipmVxXeNRAABzQ2bhrQM94nTDSQCJMmOg== 2024-04-26T01:07:23.903 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.5] 2024-04-26T01:07:23.903 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAj/ipmzgD+ABAAJgIrgxJDPCKVRZlPLgLKow== 2024-04-26T01:07:23.904 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.9] 2024-04-26T01:07:23.904 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAm/ipmrzk0CBAA+xbOyJAwcvMl4yZdsr+LTg== 2024-04-26T01:07:23.904 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.13] 2024-04-26T01:07:23.904 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAp/ipmD3CuDhAAGKLWUtWdZC0yZ85NVFHTTA== 2024-04-26T01:07:23.904 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.2] 2024-04-26T01:07:23.904 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAv/ipmDgUEDxAAi/EW7aDj81VCuuOMhMvqYQ== 2024-04-26T01:07:23.904 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.6] 2024-04-26T01:07:23.904 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAy/ipmTUp9FBAANkjvGJlyN2gEVH+lwi5U3Q== 2024-04-26T01:07:23.904 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.10] 2024-04-26T01:07:23.904 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA1/ipmV77gGhAAG8xnsr3DlF8dyvMdxDgJrg== 2024-04-26T01:07:23.904 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.14] 2024-04-26T01:07:23.904 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA4/ipmoFEAIhAA0m06wDSzUwpXear3fBIQcw== 2024-04-26T01:07:23.904 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr.x] 2024-04-26T01:07:23.904 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAP/ipms+dhHBAA8qvdsjdi2bG0e5qjHIBkNg== 2024-04-26T01:07:23.905 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.3] 2024-04-26T01:07:23.905 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA+/ipmzs5zJxAAmMSRVGcv8OMdpB5E5amXrw== 2024-04-26T01:07:23.905 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.7] 2024-04-26T01:07:23.905 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBB/ipmNiVJLRAAb7mQrRcGo9VJdCvTozWpug== 2024-04-26T01:07:23.905 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.11] 2024-04-26T01:07:23.905 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBE/ipmZ/H2MxAAFDP+GcrxtiLgE38sspRfmg== 2024-04-26T01:07:23.905 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.15] 2024-04-26T01:07:23.905 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBH/ipmd1UdOhAAJ9rhb7VNVFKKvF4m7q3gBw== 2024-04-26T01:07:23.905 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2024-04-26T01:07:23.905 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAP/ipmINz3IRAA4iN718g6n9G90IaxjaT2mg== 2024-04-26T01:07:23.907 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.y] 2024-04-26T01:07:23.907 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAP/ipmF618GRAAzhrROMUTWs51cvxMyRU5jw== 2024-04-26T01:07:23.907 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.0] 2024-04-26T01:07:23.907 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAQ/ipmaY87JhAAFRHHgcfZSG555xjSSJQDTQ== 2024-04-26T01:07:23.907 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.4] 2024-04-26T01:07:23.907 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAT/ipmQ/m+KhAAGaq0s4Hh7LbMALmtlqet3Q== 2024-04-26T01:07:23.908 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.8] 2024-04-26T01:07:23.908 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAW/ipmIh5/MRAAMqDYtjGww0fOjieYaL5iOQ== 2024-04-26T01:07:23.908 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.12] 2024-04-26T01:07:23.908 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAZ/ipmhSFuORAApFFyFZuxqyILqpfis2MQPA== 2024-04-26T01:07:23.908 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.1] 2024-04-26T01:07:23.908 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAf/ipmVxXeNRAABzQ2bhrQM94nTDSQCJMmOg== 2024-04-26T01:07:23.908 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.5] 2024-04-26T01:07:23.908 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAj/ipmzgD+ABAAJgIrgxJDPCKVRZlPLgLKow== 2024-04-26T01:07:23.908 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.9] 2024-04-26T01:07:23.908 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAm/ipmrzk0CBAA+xbOyJAwcvMl4yZdsr+LTg== 2024-04-26T01:07:23.908 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.13] 2024-04-26T01:07:23.909 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAp/ipmD3CuDhAAGKLWUtWdZC0yZ85NVFHTTA== 2024-04-26T01:07:23.909 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.2] 2024-04-26T01:07:23.909 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAv/ipmDgUEDxAAi/EW7aDj81VCuuOMhMvqYQ== 2024-04-26T01:07:23.909 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.6] 2024-04-26T01:07:23.909 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAy/ipmTUp9FBAANkjvGJlyN2gEVH+lwi5U3Q== 2024-04-26T01:07:23.909 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.10] 2024-04-26T01:07:23.909 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQA1/ipmV77gGhAAG8xnsr3DlF8dyvMdxDgJrg== 2024-04-26T01:07:23.909 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.14] 2024-04-26T01:07:23.909 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQA4/ipmoFEAIhAA0m06wDSzUwpXear3fBIQcw== 2024-04-26T01:07:23.909 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.x] 2024-04-26T01:07:23.909 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAP/ipms+dhHBAA8qvdsjdi2bG0e5qjHIBkNg== 2024-04-26T01:07:23.909 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.3] 2024-04-26T01:07:23.910 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQA+/ipmzs5zJxAAmMSRVGcv8OMdpB5E5amXrw== 2024-04-26T01:07:23.910 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.7] 2024-04-26T01:07:23.910 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBB/ipmNiVJLRAAb7mQrRcGo9VJdCvTozWpug== 2024-04-26T01:07:23.910 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.11] 2024-04-26T01:07:23.910 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBE/ipmZ/H2MxAAFDP+GcrxtiLgE38sspRfmg== 2024-04-26T01:07:23.910 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.15] 2024-04-26T01:07:23.910 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBH/ipmd1UdOhAAJ9rhb7VNVFKKvF4m7q3gBw== 2024-04-26T01:07:23.910 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-04-26T01:07:23.916 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAP/ipmINz3IRAA4iN718g6n9G90IaxjaT2mg== 2024-04-26T01:07:23.917 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T01:07:23.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T01:07:23.955 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T01:07:24.000 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.040 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.048 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.093 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.128 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.135 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.177 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.212 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.219 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.261 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.297 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.304 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.348 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.380 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.387 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.436 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.464 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.475 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.520 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.548 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.564 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.608 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.632 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.648 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.691 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.720 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.722 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.767 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.804 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.807 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.848 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.888 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.891 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.935 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.972 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:24.975 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:25.017 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T01:07:25.056 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T01:07:25.063 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T01:07:25.102 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:25.140 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:25.147 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:25.188 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:25.224 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:25.231 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:25.272 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:25.308 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:25.316 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:25.363 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:25.392 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:25.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:07:25.444 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T01:07:25.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T01:07:25.449 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T01:07:25.530 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-26T01:07:25.530 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-26T01:07:25.545 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T01:07:25.633 DEBUG:teuthology.orchestra.run.smithi008:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-26T01:07:25.644 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-26T01:07:25.660 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T01:07:25.756 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-26T01:07:25.768 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-26T01:07:25.782 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T01:07:25.876 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-26T01:07:25.887 DEBUG:teuthology.orchestra.run.smithi008:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:07:25.889 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:07:25.891 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:07:25.932 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-26T01:07:25.933 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T01:07:25.933 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T01:07:25.938 INFO:tasks.ceph.mon.a:Started 2024-04-26T01:07:25.939 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T01:07:25.939 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T01:07:25.941 INFO:tasks.ceph.mon.b:Started 2024-04-26T01:07:25.941 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T01:07:25.941 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T01:07:25.978 INFO:tasks.ceph.mon.c:Started 2024-04-26T01:07:25.978 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-26T01:07:25.978 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-26T01:07:25.979 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-26T01:07:25.981 INFO:tasks.ceph.mgr.y:Started 2024-04-26T01:07:25.981 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-26T01:07:25.981 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-26T01:07:25.984 INFO:tasks.ceph.mgr.x:Started 2024-04-26T01:07:25.984 INFO:tasks.ceph:Setting crush tunables to default 2024-04-26T01:07:25.984 DEBUG:teuthology.orchestra.run.smithi008:> sudo ceph --cluster ceph osd crush tunables default 2024-04-26T01:07:31.486 INFO:teuthology.orchestra.run.smithi008.stderr:adjusted tunables profile to default 2024-04-26T01:07:31.506 INFO:tasks.ceph:check_enable_crimson: False 2024-04-26T01:07:31.506 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-26T01:07:31.506 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:07:31.506 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-26T01:07:31.519 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:07:31.519 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-26T01:07:31.576 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:07:31.576 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-04-26T01:07:31.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:31.609+0000 7f84e8ae1180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:07:31.632 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-26T01:07:31.633 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-04-26T01:07:31.678 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:31.673+0000 7f533c8de180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:07:31.688 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:07:31.688 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-26T01:07:31.703 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:07:31.703 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-26T01:07:31.758 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:07:31.758 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-04-26T01:07:31.814 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-26T01:07:31.814 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-04-26T01:07:31.871 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:07:31.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-26T01:07:31.886 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:07:31.886 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-26T01:07:31.940 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:07:31.940 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-04-26T01:07:31.997 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-26T01:07:31.998 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-04-26T01:07:32.052 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:07:32.053 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-26T01:07:32.065 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:07:32.065 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-26T01:07:32.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:32.101+0000 7f84e8ae1180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:07:32.115 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:07:32.115 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-04-26T01:07:32.168 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-26T01:07:32.168 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-04-26T01:07:32.200 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:32.197+0000 7f533c8de180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:07:32.221 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new abed240a-e896-459b-8a43-cd1419cafbc2 0 2024-04-26T01:07:32.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:32.309+0000 7f84e8ae1180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:07:32.405 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:32.401+0000 7f533c8de180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:07:32.472 INFO:teuthology.orchestra.run.smithi164.stdout:0 2024-04-26T01:07:32.489 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 831c717c-e124-4489-a44f-93db78c47deb 1 2024-04-26T01:07:32.700 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-04-26T01:07:32.718 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 92a95165-ae34-4f29-a7f3-fe97afc26af7 2 2024-04-26T01:07:32.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:32.865+0000 7f84e8ae1180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:07:32.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:32.921+0000 7f84e8ae1180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:07:32.924 INFO:teuthology.orchestra.run.smithi164.stdout:2 2024-04-26T01:07:32.940 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new e9d8f764-4951-4de4-92c7-682850852e85 3 2024-04-26T01:07:32.977 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:32.973+0000 7f84e8ae1180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:07:32.991 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:32.989+0000 7f533c8de180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:07:33.047 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:33.045+0000 7f533c8de180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:07:33.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:33.053+0000 7f84e8ae1180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:07:33.103 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:33.101+0000 7f533c8de180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:07:33.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:33.109+0000 7f84e8ae1180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:07:33.141 INFO:teuthology.orchestra.run.smithi164.stdout:3 2024-04-26T01:07:33.157 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 7f5e93f8-b1f6-41e7-8da9-b8e57d4259bc 4 2024-04-26T01:07:33.184 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:33.181+0000 7f533c8de180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:07:33.189 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:33.185+0000 7f84e8ae1180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:07:33.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:33.233+0000 7f533c8de180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:07:33.243 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:33.241+0000 7f84e8ae1180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:07:33.319 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:33.317+0000 7f533c8de180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:07:33.375 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:33.373+0000 7f533c8de180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:07:33.382 INFO:teuthology.orchestra.run.smithi164.stdout:4 2024-04-26T01:07:33.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:33.393+0000 7f84e8ae1180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:07:33.399 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new f7b3a311-3bf6-4ae5-b7d2-dedbbe6c47ea 5 2024-04-26T01:07:33.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:33.457+0000 7f84e8ae1180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:07:33.532 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:33.529+0000 7f533c8de180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:07:33.597 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:33.593+0000 7f533c8de180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:07:33.598 INFO:teuthology.orchestra.run.smithi164.stdout:5 2024-04-26T01:07:33.613 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 796d7212-13de-43b2-8aa4-4abc4fd210a0 6 2024-04-26T01:07:33.865 INFO:teuthology.orchestra.run.smithi164.stdout:6 2024-04-26T01:07:33.882 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new ccb8cd0f-a0d6-44d7-86cc-809eff9687ce 7 2024-04-26T01:07:34.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-26T01:07:34.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-26T01:07:34.043 INFO:tasks.ceph.mgr.x.smithi164.stderr: from numpy import show_config as show_numpy_config 2024-04-26T01:07:34.078 INFO:teuthology.orchestra.run.smithi164.stdout:7 2024-04-26T01:07:34.094 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new bf13e311-71b7-44ff-af1b-68d118935c39 8 2024-04-26T01:07:34.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:34.101+0000 7f84e8ae1180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:07:34.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:34.257+0000 7f84e8ae1180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:07:34.274 INFO:tasks.ceph.mgr.y.smithi008.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-26T01:07:34.274 INFO:tasks.ceph.mgr.y.smithi008.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-26T01:07:34.274 INFO:tasks.ceph.mgr.y.smithi008.stderr: from numpy import show_config as show_numpy_config 2024-04-26T01:07:34.299 INFO:teuthology.orchestra.run.smithi164.stdout:8 2024-04-26T01:07:34.315 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 4864f46f-fd7b-48fc-908b-2564f7bda67b 9 2024-04-26T01:07:34.328 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:34.325+0000 7f533c8de180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:07:34.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:34.413+0000 7f84e8ae1180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:07:34.480 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:34.477+0000 7f533c8de180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:07:34.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:34.481+0000 7f84e8ae1180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:07:34.558 INFO:teuthology.orchestra.run.smithi164.stdout:9 2024-04-26T01:07:34.575 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 09cbb9ca-f609-409e-821b-93e695b87067 10 2024-04-26T01:07:34.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:34.617+0000 7f84e8ae1180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:07:34.649 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:34.645+0000 7f533c8de180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:07:34.716 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:34.713+0000 7f533c8de180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:07:34.779 INFO:teuthology.orchestra.run.smithi164.stdout:10 2024-04-26T01:07:34.796 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 9bfdf353-5174-40dd-bb3c-ab2dc850adad 11 2024-04-26T01:07:34.857 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:34.853+0000 7f533c8de180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:07:34.988 INFO:teuthology.orchestra.run.smithi164.stdout:11 2024-04-26T01:07:35.005 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new a3e244d8-a765-467e-820c-caa5d1d97c9f 12 2024-04-26T01:07:35.206 INFO:teuthology.orchestra.run.smithi164.stdout:12 2024-04-26T01:07:35.223 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 383875c2-0a0c-4834-8beb-a6784eef1f9f 13 2024-04-26T01:07:35.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:35.257+0000 7f84e8ae1180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:07:35.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:35.333+0000 7f84e8ae1180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:07:35.420 INFO:teuthology.orchestra.run.smithi164.stdout:13 2024-04-26T01:07:35.437 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 30311d57-c625-4dff-98d5-1ba0a84c0ceb 14 2024-04-26T01:07:35.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:35.517+0000 7f84e8ae1180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:07:35.527 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:35.525+0000 7f533c8de180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:07:35.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:35.593+0000 7f84e8ae1180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:07:35.613 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:35.609+0000 7f533c8de180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:07:35.671 INFO:teuthology.orchestra.run.smithi164.stdout:14 2024-04-26T01:07:35.687 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 1243478f-9908-460d-88d8-3c542d5258b0 15 2024-04-26T01:07:35.814 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:35.809+0000 7f533c8de180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:07:35.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:35.885+0000 7f84e8ae1180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:07:35.947 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:35.889+0000 7f533c8de180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:07:35.948 INFO:teuthology.orchestra.run.smithi164.stdout:15 2024-04-26T01:07:35.949 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-26T01:07:35.949 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-26T01:07:35.952 INFO:tasks.ceph.osd.0:Started 2024-04-26T01:07:35.952 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-26T01:07:35.952 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-26T01:07:35.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:35.949+0000 7f84e8ae1180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:07:35.955 INFO:tasks.ceph.osd.4:Started 2024-04-26T01:07:35.955 INFO:tasks.ceph.osd.8:Restarting daemon 2024-04-26T01:07:35.955 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-04-26T01:07:35.958 INFO:tasks.ceph.osd.8:Started 2024-04-26T01:07:35.958 INFO:tasks.ceph.osd.12:Restarting daemon 2024-04-26T01:07:35.958 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-04-26T01:07:35.960 INFO:tasks.ceph.osd.12:Started 2024-04-26T01:07:35.960 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-26T01:07:35.960 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-26T01:07:35.963 INFO:tasks.ceph.osd.1:Started 2024-04-26T01:07:35.963 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-26T01:07:35.963 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-26T01:07:35.966 INFO:tasks.ceph.osd.5:Started 2024-04-26T01:07:35.966 INFO:tasks.ceph.osd.9:Restarting daemon 2024-04-26T01:07:35.966 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-04-26T01:07:35.969 INFO:tasks.ceph.osd.9:Started 2024-04-26T01:07:35.969 INFO:tasks.ceph.osd.13:Restarting daemon 2024-04-26T01:07:35.969 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-04-26T01:07:35.973 INFO:tasks.ceph.osd.13:Started 2024-04-26T01:07:35.973 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-26T01:07:35.973 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-26T01:07:35.979 INFO:tasks.ceph.osd.2:Started 2024-04-26T01:07:35.979 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-26T01:07:35.979 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-26T01:07:35.982 INFO:tasks.ceph.osd.6:Started 2024-04-26T01:07:35.983 INFO:tasks.ceph.osd.10:Restarting daemon 2024-04-26T01:07:35.983 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-04-26T01:07:35.988 INFO:tasks.ceph.osd.10:Started 2024-04-26T01:07:35.988 INFO:tasks.ceph.osd.14:Restarting daemon 2024-04-26T01:07:35.989 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-04-26T01:07:35.991 INFO:tasks.ceph.osd.14:Started 2024-04-26T01:07:35.992 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-26T01:07:35.992 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-26T01:07:35.994 INFO:tasks.ceph.osd.3:Started 2024-04-26T01:07:35.994 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-26T01:07:35.995 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-26T01:07:35.996 INFO:tasks.ceph.osd.7:Started 2024-04-26T01:07:35.997 INFO:tasks.ceph.osd.11:Restarting daemon 2024-04-26T01:07:35.997 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-04-26T01:07:35.999 INFO:tasks.ceph.osd.11:Started 2024-04-26T01:07:35.999 INFO:tasks.ceph.osd.15:Restarting daemon 2024-04-26T01:07:36.000 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-04-26T01:07:36.002 INFO:tasks.ceph.osd.15:Started 2024-04-26T01:07:36.002 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T01:07:36.198 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:07:36.198 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":18,"fsid":"4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554","created":"2024-04-26T01:07:31.350368+0000","modified":"2024-04-26T01:07:35.921250+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T01:07:35.921250+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"abed240a-e896-459b-8a43-cd1419cafbc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"831c717c-e124-4489-a44f-93db78c47deb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"92a95165-ae34-4f29-a7f3-fe97afc26af7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e9d8f764-4951-4de4-92c7-682850852e85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7f5e93f8-b1f6-41e7-8da9-b8e57d4259bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f7b3a311-3bf6-4ae5-b7d2-dedbbe6c47ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"796d7212-13de-43b2-8aa4-4abc4fd210a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ccb8cd0f-a0d6-44d7-86cc-809eff9687ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"bf13e311-71b7-44ff-af1b-68d118935c39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"4864f46f-fd7b-48fc-908b-2564f7bda67b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"09cbb9ca-f609-409e-821b-93e695b87067","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9bfdf353-5174-40dd-bb3c-ab2dc850adad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"a3e244d8-a765-467e-820c-caa5d1d97c9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"383875c2-0a0c-4834-8beb-a6784eef1f9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"30311d57-c625-4dff-98d5-1ba0a84c0ceb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"1243478f-9908-460d-88d8-3c542d5258b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:07:36.219 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-26T01:07:36.219 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-26T01:07:36.230 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:36.225+0000 7f533c8de180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:07:36.295 INFO:tasks.ceph.mgr.y.smithi008.stderr:2024-04-26T01:07:36.293+0000 7f533c8de180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:07:36.576 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:07:36.573+0000 7fba22a937c0 -1 Falling back to public interface 2024-04-26T01:07:36.583 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:07:36.581+0000 7f2dbfb377c0 -1 Falling back to public interface 2024-04-26T01:07:36.583 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:07:36.581+0000 7f1c2f3f57c0 -1 Falling back to public interface 2024-04-26T01:07:36.595 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:07:36.593+0000 7f6ef7ce07c0 -1 Falling back to public interface 2024-04-26T01:07:36.604 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:07:36.601+0000 7fba1500e7c0 -1 Falling back to public interface 2024-04-26T01:07:36.604 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:07:36.601+0000 7f0ae3d7a7c0 -1 Falling back to public interface 2024-04-26T01:07:36.611 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:07:36.609+0000 7efd40b307c0 -1 Falling back to public interface 2024-04-26T01:07:36.615 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:07:36.613+0000 7f687c86b7c0 -1 Falling back to public interface 2024-04-26T01:07:36.615 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:07:36.613+0000 7f6ad481f7c0 -1 Falling back to public interface 2024-04-26T01:07:36.616 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:07:36.613+0000 7f7183dac7c0 -1 Falling back to public interface 2024-04-26T01:07:36.622 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:07:36.620+0000 7f7d665667c0 -1 Falling back to public interface 2024-04-26T01:07:36.622 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:07:36.620+0000 7fe1b57a37c0 -1 Falling back to public interface 2024-04-26T01:07:36.626 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:07:36.625+0000 7f3bc682e7c0 -1 Falling back to public interface 2024-04-26T01:07:36.626 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:07:36.625+0000 7f5c972547c0 -1 Falling back to public interface 2024-04-26T01:07:36.641 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:07:36.640+0000 7f67bebe77c0 -1 Falling back to public interface 2024-04-26T01:07:36.642 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:07:36.640+0000 7f0f709547c0 -1 Falling back to public interface 2024-04-26T01:07:36.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:36.993+0000 7f8490029640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-26T01:07:37.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:07:37.001+0000 7f8490029640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-26T01:07:37.827 DEBUG:teuthology.orchestra.run.smithi008:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T01:07:38.125 INFO:teuthology.misc.health.smithi008.stdout: 2024-04-26T01:07:38.125 INFO:teuthology.misc.health.smithi008.stdout:{"epoch":18,"fsid":"4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554","created":"2024-04-26T01:07:31.350368+0000","modified":"2024-04-26T01:07:35.921250+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T01:07:35.921250+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"abed240a-e896-459b-8a43-cd1419cafbc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"831c717c-e124-4489-a44f-93db78c47deb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"92a95165-ae34-4f29-a7f3-fe97afc26af7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e9d8f764-4951-4de4-92c7-682850852e85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7f5e93f8-b1f6-41e7-8da9-b8e57d4259bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f7b3a311-3bf6-4ae5-b7d2-dedbbe6c47ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"796d7212-13de-43b2-8aa4-4abc4fd210a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ccb8cd0f-a0d6-44d7-86cc-809eff9687ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"bf13e311-71b7-44ff-af1b-68d118935c39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"4864f46f-fd7b-48fc-908b-2564f7bda67b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"09cbb9ca-f609-409e-821b-93e695b87067","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9bfdf353-5174-40dd-bb3c-ab2dc850adad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"a3e244d8-a765-467e-820c-caa5d1d97c9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"383875c2-0a0c-4834-8beb-a6784eef1f9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"30311d57-c625-4dff-98d5-1ba0a84c0ceb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"1243478f-9908-460d-88d8-3c542d5258b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:07:38.142 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-04-26T01:07:40.316 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:07:40.313+0000 7f2dbfb377c0 -1 osd.12 0 log_to_monitors true 2024-04-26T01:07:40.330 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:07:40.328+0000 7f7d665667c0 -1 osd.14 0 log_to_monitors true 2024-04-26T01:07:40.378 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:07:40.377+0000 7f0ae3d7a7c0 -1 osd.15 0 log_to_monitors true 2024-04-26T01:07:40.395 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:07:40.393+0000 7f6ad481f7c0 -1 osd.5 0 log_to_monitors true 2024-04-26T01:07:40.408 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:07:40.405+0000 7fba22a937c0 -1 osd.0 0 log_to_monitors true 2024-04-26T01:07:40.448 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:07:40.445+0000 7f687c86b7c0 -1 osd.11 0 log_to_monitors true 2024-04-26T01:07:40.464 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:07:40.461+0000 7f6ef7ce07c0 -1 osd.4 0 log_to_monitors true 2024-04-26T01:07:40.468 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:07:40.464+0000 7f0f709547c0 -1 osd.6 0 log_to_monitors true 2024-04-26T01:07:40.470 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:07:40.469+0000 7fba1500e7c0 -1 osd.3 0 log_to_monitors true 2024-04-26T01:07:40.471 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:07:40.468+0000 7fe1b57a37c0 -1 osd.2 0 log_to_monitors true 2024-04-26T01:07:40.481 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:07:40.481+0000 7efd40b307c0 -1 osd.9 0 log_to_monitors true 2024-04-26T01:07:40.493 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:07:40.489+0000 7f3bc682e7c0 -1 osd.1 0 log_to_monitors true 2024-04-26T01:07:40.493 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:07:40.493+0000 7f5c972547c0 -1 osd.13 0 log_to_monitors true 2024-04-26T01:07:40.516 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:07:40.513+0000 7f1c2f3f57c0 -1 osd.8 0 log_to_monitors true 2024-04-26T01:07:40.521 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:07:40.520+0000 7f67bebe77c0 -1 osd.10 0 log_to_monitors true 2024-04-26T01:07:40.538 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:07:40.537+0000 7f7183dac7c0 -1 osd.7 0 log_to_monitors true 2024-04-26T01:07:42.020 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:07:42.016+0000 7f7d62d21640 -1 osd.14 0 waiting for initial osdmap 2024-04-26T01:07:42.022 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:07:42.020+0000 7f67bab90640 -1 osd.10 0 waiting for initial osdmap 2024-04-26T01:07:42.026 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:07:42.025+0000 7f6ef3c89640 -1 osd.4 0 waiting for initial osdmap 2024-04-26T01:07:42.027 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:07:42.025+0000 7f1c2b39e640 -1 osd.8 0 waiting for initial osdmap 2024-04-26T01:07:42.035 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:07:42.033+0000 7f717fd55640 -1 osd.7 0 waiting for initial osdmap 2024-04-26T01:07:42.035 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:07:42.033+0000 7efd3cad9640 -1 osd.9 0 waiting for initial osdmap 2024-04-26T01:07:42.041 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:07:42.040+0000 7f7d5e34c640 -1 osd.14 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:42.046 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:07:42.045+0000 7f6eefac6640 -1 osd.4 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:42.049 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:07:42.045+0000 7f1c271db640 -1 osd.8 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:42.050 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:07:42.049+0000 7efd38916640 -1 osd.9 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:42.053 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:07:42.052+0000 7f67b69cd640 -1 osd.10 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:42.079 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:07:42.077+0000 7f717bb92640 -1 osd.7 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:42.811 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:07:42.809+0000 7fba1ea3c640 -1 osd.0 0 waiting for initial osdmap 2024-04-26T01:07:42.819 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:07:42.817+0000 7f2dbbae0640 -1 osd.12 0 waiting for initial osdmap 2024-04-26T01:07:42.829 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:07:42.828+0000 7fe1b174c640 -1 osd.2 0 waiting for initial osdmap 2024-04-26T01:07:42.830 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:07:42.829+0000 7fba16871640 -1 osd.0 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:42.851 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:07:42.849+0000 7f2db3915640 -1 osd.12 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:42.857 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:07:42.856+0000 7fe1a9581640 -1 osd.2 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:42.858 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:07:42.856+0000 7f0f6c8fd640 -1 osd.6 0 waiting for initial osdmap 2024-04-26T01:07:42.893 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:07:42.892+0000 7f0f64732640 -1 osd.6 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:42.922 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:07:42.920+0000 7f0adfd23640 -1 osd.15 0 waiting for initial osdmap 2024-04-26T01:07:42.947 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:07:42.944+0000 7f0ad7b58640 -1 osd.15 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:42.986 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:07:42.984+0000 7f6879026640 -1 osd.11 0 waiting for initial osdmap 2024-04-26T01:07:43.008 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:07:43.004+0000 7fba10fb7640 -1 osd.3 0 waiting for initial osdmap 2024-04-26T01:07:43.015 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:07:43.012+0000 7f6870649640 -1 osd.11 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:43.035 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:07:43.032+0000 7fba08dec640 -1 osd.3 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:43.089 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:07:43.085+0000 7f3bc27d7640 -1 osd.1 0 waiting for initial osdmap 2024-04-26T01:07:43.093 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:07:43.093+0000 7f5c93a0f640 -1 osd.13 0 waiting for initial osdmap 2024-04-26T01:07:43.105 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:07:43.101+0000 7f6ad0fda640 -1 osd.5 0 waiting for initial osdmap 2024-04-26T01:07:43.114 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:07:43.113+0000 7f3bba60c640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:43.133 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:07:43.133+0000 7f5c8b032640 -1 osd.13 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:43.142 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:07:43.141+0000 7f6ac85fd640 -1 osd.5 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:07:45.749 DEBUG:teuthology.orchestra.run.smithi008:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T01:07:46.063 INFO:teuthology.misc.health.smithi008.stdout: 2024-04-26T01:07:46.063 INFO:teuthology.misc.health.smithi008.stdout:{"epoch":24,"fsid":"4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554","created":"2024-04-26T01:07:31.350368+0000","modified":"2024-04-26T01:07:46.034187+0000","last_up_change":"2024-04-26T01:07:44.028125+0000","last_in_change":"2024-04-26T01:07:35.921250+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:07:44.085247+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"abed240a-e896-459b-8a43-cd1419cafbc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6801","nonce":188308042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6803","nonce":188308042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6807","nonce":188308042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6805","nonce":188308042}]},"public_addr":"172.21.15.8:6801/188308042","cluster_addr":"172.21.15.8:6803/188308042","heartbeat_back_addr":"172.21.15.8:6807/188308042","heartbeat_front_addr":"172.21.15.8:6805/188308042","state":["exists","up"]},{"osd":1,"uuid":"831c717c-e124-4489-a44f-93db78c47deb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6819","nonce":3487767810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6823","nonce":3487767810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6831","nonce":3487767810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6828","nonce":3487767810}]},"public_addr":"172.21.15.12:6819/3487767810","cluster_addr":"172.21.15.12:6823/3487767810","heartbeat_back_addr":"172.21.15.12:6831/3487767810","heartbeat_front_addr":"172.21.15.12:6828/3487767810","state":["exists","up"]},{"osd":2,"uuid":"92a95165-ae34-4f29-a7f3-fe97afc26af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6803","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6805","nonce":3599623558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6807","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6809","nonce":3599623558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6815","nonce":3599623558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6813","nonce":3599623558}]},"public_addr":"172.21.15.116:6805/3599623558","cluster_addr":"172.21.15.116:6809/3599623558","heartbeat_back_addr":"172.21.15.116:6815/3599623558","heartbeat_front_addr":"172.21.15.116:6813/3599623558","state":["exists","up"]},{"osd":3,"uuid":"e9d8f764-4951-4de4-92c7-682850852e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6804","nonce":4123918458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6808","nonce":4123918458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6816","nonce":4123918458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6812","nonce":4123918458}]},"public_addr":"172.21.15.164:6804/4123918458","cluster_addr":"172.21.15.164:6808/4123918458","heartbeat_back_addr":"172.21.15.164:6816/4123918458","heartbeat_front_addr":"172.21.15.164:6812/4123918458","state":["exists","up"]},{"osd":4,"uuid":"7f5e93f8-b1f6-41e7-8da9-b8e57d4259bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6825","nonce":425515720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6827","nonce":425515720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6831","nonce":425515720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6829","nonce":425515720}]},"public_addr":"172.21.15.8:6825/425515720","cluster_addr":"172.21.15.8:6827/425515720","heartbeat_back_addr":"172.21.15.8:6831/425515720","heartbeat_front_addr":"172.21.15.8:6829/425515720","state":["exists","up"]},{"osd":5,"uuid":"f7b3a311-3bf6-4ae5-b7d2-dedbbe6c47ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6809","nonce":1641884791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6811","nonce":1641884791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6815","nonce":1641884791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6813","nonce":1641884791}]},"public_addr":"172.21.15.12:6809/1641884791","cluster_addr":"172.21.15.12:6811/1641884791","heartbeat_back_addr":"172.21.15.12:6815/1641884791","heartbeat_front_addr":"172.21.15.12:6813/1641884791","state":["exists","up"]},{"osd":6,"uuid":"796d7212-13de-43b2-8aa4-4abc4fd210a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6817","nonce":156763037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6820","nonce":156763037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6825","nonce":156763037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6821","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6823","nonce":156763037}]},"public_addr":"172.21.15.116:6817/156763037","cluster_addr":"172.21.15.116:6820/156763037","heartbeat_back_addr":"172.21.15.116:6825/156763037","heartbeat_front_addr":"172.21.15.116:6823/156763037","state":["exists","up"]},{"osd":7,"uuid":"ccb8cd0f-a0d6-44d7-86cc-809eff9687ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6826","nonce":2291946233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6829","nonce":2291946233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6833","nonce":2291946233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6831","nonce":2291946233}]},"public_addr":"172.21.15.164:6826/2291946233","cluster_addr":"172.21.15.164:6829/2291946233","heartbeat_back_addr":"172.21.15.164:6833/2291946233","heartbeat_front_addr":"172.21.15.164:6831/2291946233","state":["exists","up"]},{"osd":8,"uuid":"bf13e311-71b7-44ff-af1b-68d118935c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6809","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6811","nonce":2709504491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6813","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6815","nonce":2709504491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6821","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6823","nonce":2709504491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6817","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6819","nonce":2709504491}]},"public_addr":"172.21.15.8:6811/2709504491","cluster_addr":"172.21.15.8:6815/2709504491","heartbeat_back_addr":"172.21.15.8:6823/2709504491","heartbeat_front_addr":"172.21.15.8:6819/2709504491","state":["exists","up"]},{"osd":9,"uuid":"4864f46f-fd7b-48fc-908b-2564f7bda67b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6801","nonce":1155130892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6803","nonce":1155130892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6807","nonce":1155130892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6805","nonce":1155130892}]},"public_addr":"172.21.15.12:6801/1155130892","cluster_addr":"172.21.15.12:6803/1155130892","heartbeat_back_addr":"172.21.15.12:6807/1155130892","heartbeat_front_addr":"172.21.15.12:6805/1155130892","state":["exists","up"]},{"osd":10,"uuid":"09cbb9ca-f609-409e-821b-93e695b87067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6822","nonce":1461831728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6827","nonce":1461831728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6831","nonce":1461831728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6829","nonce":1461831728}]},"public_addr":"172.21.15.116:6822/1461831728","cluster_addr":"172.21.15.116:6827/1461831728","heartbeat_back_addr":"172.21.15.116:6831/1461831728","heartbeat_front_addr":"172.21.15.116:6829/1461831728","state":["exists","up"]},{"osd":11,"uuid":"9bfdf353-5174-40dd-bb3c-ab2dc850adad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":1282808378},{"type":"v1","addr":"172.21.15.164:6819","nonce":1282808378}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":1282808378},{"type":"v1","addr":"172.21.15.164:6821","nonce":1282808378}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6825","nonce":1282808378},{"type":"v1","addr":"172.21.15.164:6827","nonce":1282808378}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":1282808378},{"type":"v1","addr":"172.21.15.164:6823","nonce":1282808378}]},"public_addr":"172.21.15.164:6819/1282808378","cluster_addr":"172.21.15.164:6821/1282808378","heartbeat_back_addr":"172.21.15.164:6827/1282808378","heartbeat_front_addr":"172.21.15.164:6823/1282808378","state":["exists","up"]},{"osd":12,"uuid":"a3e244d8-a765-467e-820c-caa5d1d97c9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6810","nonce":2082773890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6814","nonce":2082773890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6822","nonce":2082773890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6818","nonce":2082773890}]},"public_addr":"172.21.15.8:6810/2082773890","cluster_addr":"172.21.15.8:6814/2082773890","heartbeat_back_addr":"172.21.15.8:6822/2082773890","heartbeat_front_addr":"172.21.15.8:6818/2082773890","state":["exists","up"]},{"osd":13,"uuid":"383875c2-0a0c-4834-8beb-a6784eef1f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6817","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6818","nonce":3158583580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6822","nonce":3158583580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6829","nonce":3158583580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6826","nonce":3158583580}]},"public_addr":"172.21.15.12:6818/3158583580","cluster_addr":"172.21.15.12:6822/3158583580","heartbeat_back_addr":"172.21.15.12:6829/3158583580","heartbeat_front_addr":"172.21.15.12:6826/3158583580","state":["exists","up"]},{"osd":14,"uuid":"30311d57-c625-4dff-98d5-1ba0a84c0ceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6801","nonce":2158650188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6804","nonce":2158650188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6812","nonce":2158650188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6808","nonce":2158650188}]},"public_addr":"172.21.15.116:6801/2158650188","cluster_addr":"172.21.15.116:6804/2158650188","heartbeat_back_addr":"172.21.15.116:6812/2158650188","heartbeat_front_addr":"172.21.15.116:6808/2158650188","state":["exists","up"]},{"osd":15,"uuid":"1243478f-9908-460d-88d8-3c542d5258b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6805","nonce":2601606699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6809","nonce":2601606699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6817","nonce":2601606699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6813","nonce":2601606699}]},"public_addr":"172.21.15.164:6805/2601606699","cluster_addr":"172.21.15.164:6809/2601606699","heartbeat_back_addr":"172.21.15.164:6817/2601606699","heartbeat_front_addr":"172.21.15.164:6813/2601606699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.408900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.485708+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.437264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.524213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.474428+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.415704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.513094+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.501842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.544193+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.492788+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.473510+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.411335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.316127+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.506103+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.296055+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.338969+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:07:46.081 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-04-26T01:07:46.081 INFO:tasks.ceph:Creating RBD pool 2024-04-26T01:07:46.081 DEBUG:teuthology.orchestra.run.smithi008:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-26T01:07:47.125 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'rbd' created 2024-04-26T01:07:47.150 DEBUG:teuthology.orchestra.run.smithi008:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-26T01:07:48.133 INFO:teuthology.orchestra.run.smithi008.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-26T01:07:48.162 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-26T01:07:48.163 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T01:07:48.163 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-26T01:07:48.163 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-26T01:07:48.163 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-26T01:07:48.521 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:07:48.538 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":4,"flags":0,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2789699691},{"type":"v1","addr":"172.21.15.164:6801","nonce":2789699691}]},"active_addr":"172.21.15.164:6801/2789699691","active_change":"2024-04-26T01:07:35.971311+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":2483988485}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":1862014999}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":3390931792}]}]} 2024-04-26T01:07:48.542 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-26T01:07:48.542 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-26T01:07:48.542 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T01:07:48.854 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:07:48.854 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":26,"fsid":"4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554","created":"2024-04-26T01:07:31.350368+0000","modified":"2024-04-26T01:07:48.124917+0000","last_up_change":"2024-04-26T01:07:44.028125+0000","last_in_change":"2024-04-26T01:07:35.921250+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:07:44.085247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T01:07:46.423491+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"abed240a-e896-459b-8a43-cd1419cafbc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6801","nonce":188308042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6803","nonce":188308042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6807","nonce":188308042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6805","nonce":188308042}]},"public_addr":"172.21.15.8:6801/188308042","cluster_addr":"172.21.15.8:6803/188308042","heartbeat_back_addr":"172.21.15.8:6807/188308042","heartbeat_front_addr":"172.21.15.8:6805/188308042","state":["exists","up"]},{"osd":1,"uuid":"831c717c-e124-4489-a44f-93db78c47deb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6819","nonce":3487767810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6823","nonce":3487767810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6831","nonce":3487767810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6828","nonce":3487767810}]},"public_addr":"172.21.15.12:6819/3487767810","cluster_addr":"172.21.15.12:6823/3487767810","heartbeat_back_addr":"172.21.15.12:6831/3487767810","heartbeat_front_addr":"172.21.15.12:6828/3487767810","state":["exists","up"]},{"osd":2,"uuid":"92a95165-ae34-4f29-a7f3-fe97afc26af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6803","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6805","nonce":3599623558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6807","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6809","nonce":3599623558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6815","nonce":3599623558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6813","nonce":3599623558}]},"public_addr":"172.21.15.116:6805/3599623558","cluster_addr":"172.21.15.116:6809/3599623558","heartbeat_back_addr":"172.21.15.116:6815/3599623558","heartbeat_front_addr":"172.21.15.116:6813/3599623558","state":["exists","up"]},{"osd":3,"uuid":"e9d8f764-4951-4de4-92c7-682850852e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6804","nonce":4123918458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6808","nonce":4123918458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6816","nonce":4123918458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6812","nonce":4123918458}]},"public_addr":"172.21.15.164:6804/4123918458","cluster_addr":"172.21.15.164:6808/4123918458","heartbeat_back_addr":"172.21.15.164:6816/4123918458","heartbeat_front_addr":"172.21.15.164:6812/4123918458","state":["exists","up"]},{"osd":4,"uuid":"7f5e93f8-b1f6-41e7-8da9-b8e57d4259bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6825","nonce":425515720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6827","nonce":425515720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6831","nonce":425515720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6829","nonce":425515720}]},"public_addr":"172.21.15.8:6825/425515720","cluster_addr":"172.21.15.8:6827/425515720","heartbeat_back_addr":"172.21.15.8:6831/425515720","heartbeat_front_addr":"172.21.15.8:6829/425515720","state":["exists","up"]},{"osd":5,"uuid":"f7b3a311-3bf6-4ae5-b7d2-dedbbe6c47ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6809","nonce":1641884791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6811","nonce":1641884791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6815","nonce":1641884791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6813","nonce":1641884791}]},"public_addr":"172.21.15.12:6809/1641884791","cluster_addr":"172.21.15.12:6811/1641884791","heartbeat_back_addr":"172.21.15.12:6815/1641884791","heartbeat_front_addr":"172.21.15.12:6813/1641884791","state":["exists","up"]},{"osd":6,"uuid":"796d7212-13de-43b2-8aa4-4abc4fd210a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6817","nonce":156763037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6820","nonce":156763037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6825","nonce":156763037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6821","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6823","nonce":156763037}]},"public_addr":"172.21.15.116:6817/156763037","cluster_addr":"172.21.15.116:6820/156763037","heartbeat_back_addr":"172.21.15.116:6825/156763037","heartbeat_front_addr":"172.21.15.116:6823/156763037","state":["exists","up"]},{"osd":7,"uuid":"ccb8cd0f-a0d6-44d7-86cc-809eff9687ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6826","nonce":2291946233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6829","nonce":2291946233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6833","nonce":2291946233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6831","nonce":2291946233}]},"public_addr":"172.21.15.164:6826/2291946233","cluster_addr":"172.21.15.164:6829/2291946233","heartbeat_back_addr":"172.21.15.164:6833/2291946233","heartbeat_front_addr":"172.21.15.164:6831/2291946233","state":["exists","up"]},{"osd":8,"uuid":"bf13e311-71b7-44ff-af1b-68d118935c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6809","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6811","nonce":2709504491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6813","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6815","nonce":2709504491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6821","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6823","nonce":2709504491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6817","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6819","nonce":2709504491}]},"public_addr":"172.21.15.8:6811/2709504491","cluster_addr":"172.21.15.8:6815/2709504491","heartbeat_back_addr":"172.21.15.8:6823/2709504491","heartbeat_front_addr":"172.21.15.8:6819/2709504491","state":["exists","up"]},{"osd":9,"uuid":"4864f46f-fd7b-48fc-908b-2564f7bda67b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6801","nonce":1155130892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6803","nonce":1155130892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6807","nonce":1155130892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6805","nonce":1155130892}]},"public_addr":"172.21.15.12:6801/1155130892","cluster_addr":"172.21.15.12:6803/1155130892","heartbeat_back_addr":"172.21.15.12:6807/1155130892","heartbeat_front_addr":"172.21.15.12:6805/1155130892","state":["exists","up"]},{"osd":10,"uuid":"09cbb9ca-f609-409e-821b-93e695b87067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6822","nonce":1461831728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6827","nonce":1461831728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6831","nonce":1461831728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6829","nonce":1461831728}]},"public_addr":"172.21.15.116:6822/1461831728","cluster_addr":"172.21.15.116:6827/1461831728","heartbeat_back_addr":"172.21.15.116:6831/1461831728","heartbeat_front_addr":"172.21.15.116:6829/1461831728","state":["exists","up"]},{"osd":11,"uuid":"9bfdf353-5174-40dd-bb3c-ab2dc850adad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":1282808378},{"type":"v1","addr":"172.21.15.164:6819","nonce":1282808378}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":1282808378},{"type":"v1","addr":"172.21.15.164:6821","nonce":1282808378}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6825","nonce":1282808378},{"type":"v1","addr":"172.21.15.164:6827","nonce":1282808378}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":1282808378},{"type":"v1","addr":"172.21.15.164:6823","nonce":1282808378}]},"public_addr":"172.21.15.164:6819/1282808378","cluster_addr":"172.21.15.164:6821/1282808378","heartbeat_back_addr":"172.21.15.164:6827/1282808378","heartbeat_front_addr":"172.21.15.164:6823/1282808378","state":["exists","up"]},{"osd":12,"uuid":"a3e244d8-a765-467e-820c-caa5d1d97c9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6810","nonce":2082773890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6814","nonce":2082773890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6822","nonce":2082773890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6818","nonce":2082773890}]},"public_addr":"172.21.15.8:6810/2082773890","cluster_addr":"172.21.15.8:6814/2082773890","heartbeat_back_addr":"172.21.15.8:6822/2082773890","heartbeat_front_addr":"172.21.15.8:6818/2082773890","state":["exists","up"]},{"osd":13,"uuid":"383875c2-0a0c-4834-8beb-a6784eef1f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6817","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6818","nonce":3158583580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6822","nonce":3158583580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6829","nonce":3158583580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6826","nonce":3158583580}]},"public_addr":"172.21.15.12:6818/3158583580","cluster_addr":"172.21.15.12:6822/3158583580","heartbeat_back_addr":"172.21.15.12:6829/3158583580","heartbeat_front_addr":"172.21.15.12:6826/3158583580","state":["exists","up"]},{"osd":14,"uuid":"30311d57-c625-4dff-98d5-1ba0a84c0ceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6801","nonce":2158650188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6804","nonce":2158650188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6812","nonce":2158650188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6808","nonce":2158650188}]},"public_addr":"172.21.15.116:6801/2158650188","cluster_addr":"172.21.15.116:6804/2158650188","heartbeat_back_addr":"172.21.15.116:6812/2158650188","heartbeat_front_addr":"172.21.15.116:6808/2158650188","state":["exists","up"]},{"osd":15,"uuid":"1243478f-9908-460d-88d8-3c542d5258b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6805","nonce":2601606699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6809","nonce":2601606699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6817","nonce":2601606699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6813","nonce":2601606699}]},"public_addr":"172.21.15.164:6805/2601606699","cluster_addr":"172.21.15.164:6809/2601606699","heartbeat_back_addr":"172.21.15.164:6817/2601606699","heartbeat_front_addr":"172.21.15.164:6813/2601606699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.408900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.485708+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.437264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.524213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.474428+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.415704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.513094+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.501842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.544193+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.492788+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.473510+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.411335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.316127+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.506103+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.296055+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.338969+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:07:48.873 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-26T01:07:48.873 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T01:07:49.184 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:07:49.184 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":27,"fsid":"4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554","created":"2024-04-26T01:07:31.350368+0000","modified":"2024-04-26T01:07:49.132650+0000","last_up_change":"2024-04-26T01:07:44.028125+0000","last_in_change":"2024-04-26T01:07:35.921250+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:07:44.085247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T01:07:46.423491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"abed240a-e896-459b-8a43-cd1419cafbc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6801","nonce":188308042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6803","nonce":188308042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6807","nonce":188308042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6805","nonce":188308042}]},"public_addr":"172.21.15.8:6801/188308042","cluster_addr":"172.21.15.8:6803/188308042","heartbeat_back_addr":"172.21.15.8:6807/188308042","heartbeat_front_addr":"172.21.15.8:6805/188308042","state":["exists","up"]},{"osd":1,"uuid":"831c717c-e124-4489-a44f-93db78c47deb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6819","nonce":3487767810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6823","nonce":3487767810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6831","nonce":3487767810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6828","nonce":3487767810}]},"public_addr":"172.21.15.12:6819/3487767810","cluster_addr":"172.21.15.12:6823/3487767810","heartbeat_back_addr":"172.21.15.12:6831/3487767810","heartbeat_front_addr":"172.21.15.12:6828/3487767810","state":["exists","up"]},{"osd":2,"uuid":"92a95165-ae34-4f29-a7f3-fe97afc26af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6803","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6805","nonce":3599623558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6807","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6809","nonce":3599623558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6815","nonce":3599623558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6813","nonce":3599623558}]},"public_addr":"172.21.15.116:6805/3599623558","cluster_addr":"172.21.15.116:6809/3599623558","heartbeat_back_addr":"172.21.15.116:6815/3599623558","heartbeat_front_addr":"172.21.15.116:6813/3599623558","state":["exists","up"]},{"osd":3,"uuid":"e9d8f764-4951-4de4-92c7-682850852e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6804","nonce":4123918458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6808","nonce":4123918458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6816","nonce":4123918458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6812","nonce":4123918458}]},"public_addr":"172.21.15.164:6804/4123918458","cluster_addr":"172.21.15.164:6808/4123918458","heartbeat_back_addr":"172.21.15.164:6816/4123918458","heartbeat_front_addr":"172.21.15.164:6812/4123918458","state":["exists","up"]},{"osd":4,"uuid":"7f5e93f8-b1f6-41e7-8da9-b8e57d4259bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6825","nonce":425515720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6827","nonce":425515720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6831","nonce":425515720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6829","nonce":425515720}]},"public_addr":"172.21.15.8:6825/425515720","cluster_addr":"172.21.15.8:6827/425515720","heartbeat_back_addr":"172.21.15.8:6831/425515720","heartbeat_front_addr":"172.21.15.8:6829/425515720","state":["exists","up"]},{"osd":5,"uuid":"f7b3a311-3bf6-4ae5-b7d2-dedbbe6c47ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6809","nonce":1641884791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6811","nonce":1641884791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6815","nonce":1641884791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6813","nonce":1641884791}]},"public_addr":"172.21.15.12:6809/1641884791","cluster_addr":"172.21.15.12:6811/1641884791","heartbeat_back_addr":"172.21.15.12:6815/1641884791","heartbeat_front_addr":"172.21.15.12:6813/1641884791","state":["exists","up"]},{"osd":6,"uuid":"796d7212-13de-43b2-8aa4-4abc4fd210a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6817","nonce":156763037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6820","nonce":156763037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6825","nonce":156763037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6821","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6823","nonce":156763037}]},"public_addr":"172.21.15.116:6817/156763037","cluster_addr":"172.21.15.116:6820/156763037","heartbeat_back_addr":"172.21.15.116:6825/156763037","heartbeat_front_addr":"172.21.15.116:6823/156763037","state":["exists","up"]},{"osd":7,"uuid":"ccb8cd0f-a0d6-44d7-86cc-809eff9687ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6826","nonce":2291946233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6829","nonce":2291946233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6833","nonce":2291946233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6831","nonce":2291946233}]},"public_addr":"172.21.15.164:6826/2291946233","cluster_addr":"172.21.15.164:6829/2291946233","heartbeat_back_addr":"172.21.15.164:6833/2291946233","heartbeat_front_addr":"172.21.15.164:6831/2291946233","state":["exists","up"]},{"osd":8,"uuid":"bf13e311-71b7-44ff-af1b-68d118935c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6809","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6811","nonce":2709504491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6813","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6815","nonce":2709504491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6821","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6823","nonce":2709504491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6817","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6819","nonce":2709504491}]},"public_addr":"172.21.15.8:6811/2709504491","cluster_addr":"172.21.15.8:6815/2709504491","heartbeat_back_addr":"172.21.15.8:6823/2709504491","heartbeat_front_addr":"172.21.15.8:6819/2709504491","state":["exists","up"]},{"osd":9,"uuid":"4864f46f-fd7b-48fc-908b-2564f7bda67b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6801","nonce":1155130892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6803","nonce":1155130892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6807","nonce":1155130892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6805","nonce":1155130892}]},"public_addr":"172.21.15.12:6801/1155130892","cluster_addr":"172.21.15.12:6803/1155130892","heartbeat_back_addr":"172.21.15.12:6807/1155130892","heartbeat_front_addr":"172.21.15.12:6805/1155130892","state":["exists","up"]},{"osd":10,"uuid":"09cbb9ca-f609-409e-821b-93e695b87067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6822","nonce":1461831728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6827","nonce":1461831728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6831","nonce":1461831728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6829","nonce":1461831728}]},"public_addr":"172.21.15.116:6822/1461831728","cluster_addr":"172.21.15.116:6827/1461831728","heartbeat_back_addr":"172.21.15.116:6831/1461831728","heartbeat_front_addr":"172.21.15.116:6829/1461831728","state":["exists","up"]},{"osd":11,"uuid":"9bfdf353-5174-40dd-bb3c-ab2dc850adad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":1282808378},{"type":"v1","addr":"172.21.15.164:6819","nonce":1282808378}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":1282808378},{"type":"v1","addr":"172.21.15.164:6821","nonce":1282808378}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6825","nonce":1282808378},{"type":"v1","addr":"172.21.15.164:6827","nonce":1282808378}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":1282808378},{"type":"v1","addr":"172.21.15.164:6823","nonce":1282808378}]},"public_addr":"172.21.15.164:6819/1282808378","cluster_addr":"172.21.15.164:6821/1282808378","heartbeat_back_addr":"172.21.15.164:6827/1282808378","heartbeat_front_addr":"172.21.15.164:6823/1282808378","state":["exists","up"]},{"osd":12,"uuid":"a3e244d8-a765-467e-820c-caa5d1d97c9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6810","nonce":2082773890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6814","nonce":2082773890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6822","nonce":2082773890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6818","nonce":2082773890}]},"public_addr":"172.21.15.8:6810/2082773890","cluster_addr":"172.21.15.8:6814/2082773890","heartbeat_back_addr":"172.21.15.8:6822/2082773890","heartbeat_front_addr":"172.21.15.8:6818/2082773890","state":["exists","up"]},{"osd":13,"uuid":"383875c2-0a0c-4834-8beb-a6784eef1f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6817","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6818","nonce":3158583580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6822","nonce":3158583580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6829","nonce":3158583580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6826","nonce":3158583580}]},"public_addr":"172.21.15.12:6818/3158583580","cluster_addr":"172.21.15.12:6822/3158583580","heartbeat_back_addr":"172.21.15.12:6829/3158583580","heartbeat_front_addr":"172.21.15.12:6826/3158583580","state":["exists","up"]},{"osd":14,"uuid":"30311d57-c625-4dff-98d5-1ba0a84c0ceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6801","nonce":2158650188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6804","nonce":2158650188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6812","nonce":2158650188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6808","nonce":2158650188}]},"public_addr":"172.21.15.116:6801/2158650188","cluster_addr":"172.21.15.116:6804/2158650188","heartbeat_back_addr":"172.21.15.116:6812/2158650188","heartbeat_front_addr":"172.21.15.116:6808/2158650188","state":["exists","up"]},{"osd":15,"uuid":"1243478f-9908-460d-88d8-3c542d5258b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6805","nonce":2601606699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6809","nonce":2601606699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6817","nonce":2601606699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6813","nonce":2601606699}]},"public_addr":"172.21.15.164:6805/2601606699","cluster_addr":"172.21.15.164:6809/2601606699","heartbeat_back_addr":"172.21.15.164:6817/2601606699","heartbeat_front_addr":"172.21.15.164:6813/2601606699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.408900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.485708+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.437264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.524213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.474428+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.415704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.513094+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.501842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.544193+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.492788+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.473510+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.411335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.316127+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.506103+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.296055+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.338969+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:07:49.203 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-26T01:07:49.204 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-26T01:07:49.204 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-26T01:07:49.205 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-26T01:07:49.205 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-26T01:07:49.205 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-26T01:07:49.206 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-26T01:07:49.206 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-26T01:07:49.207 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-04-26T01:07:49.207 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-04-26T01:07:49.207 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-04-26T01:07:49.208 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-04-26T01:07:49.208 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-04-26T01:07:49.209 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-04-26T01:07:49.209 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-04-26T01:07:49.209 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-04-26T01:07:49.615 INFO:teuthology.orchestra.run.smithi008.stdout:94489280516 2024-04-26T01:07:49.616 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-26T01:07:49.652 INFO:teuthology.orchestra.run.smithi008.stdout:90194313219 2024-04-26T01:07:49.652 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-04-26T01:07:49.698 INFO:teuthology.orchestra.run.smithi008.stdout:90194313219 2024-04-26T01:07:49.698 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-04-26T01:07:49.724 INFO:teuthology.orchestra.run.smithi008.stdout:90194313221 2024-04-26T01:07:49.724 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-04-26T01:07:49.728 INFO:teuthology.orchestra.run.smithi008.stdout:90194313219 2024-04-26T01:07:49.729 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-26T01:07:49.737 INFO:teuthology.orchestra.run.smithi008.stdout:90194313220 2024-04-26T01:07:49.738 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-26T01:07:49.742 INFO:teuthology.orchestra.run.smithi008.stdout:94489280516 2024-04-26T01:07:49.743 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T01:07:49.752 INFO:teuthology.orchestra.run.smithi008.stdout:90194313220 2024-04-26T01:07:49.752 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T01:07:49.770 INFO:teuthology.orchestra.run.smithi008.stdout:90194313220 2024-04-26T01:07:49.770 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-26T01:07:49.780 INFO:teuthology.orchestra.run.smithi008.stdout:90194313219 2024-04-26T01:07:49.780 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T01:07:49.784 INFO:teuthology.orchestra.run.smithi008.stdout:90194313220 2024-04-26T01:07:49.784 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-04-26T01:07:49.833 INFO:teuthology.orchestra.run.smithi008.stdout:90194313221 2024-04-26T01:07:49.834 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-04-26T01:07:49.835 INFO:teuthology.orchestra.run.smithi008.stdout:90194313220 2024-04-26T01:07:49.835 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-04-26T01:07:49.847 INFO:teuthology.orchestra.run.smithi008.stdout:94489280515 2024-04-26T01:07:49.847 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T01:07:49.860 INFO:teuthology.orchestra.run.smithi008.stdout:94489280514 2024-04-26T01:07:49.860 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-04-26T01:07:49.912 INFO:teuthology.orchestra.run.smithi008.stdout:90194313219 2024-04-26T01:07:49.912 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-26T01:07:50.646 INFO:teuthology.orchestra.run.smithi008.stdout:90194313219 2024-04-26T01:07:50.663 INFO:teuthology.orchestra.run.smithi008.stdout:94489280516 2024-04-26T01:07:50.704 INFO:teuthology.orchestra.run.smithi008.stdout:90194313220 2024-04-26T01:07:50.705 INFO:teuthology.orchestra.run.smithi008.stdout:90194313219 2024-04-26T01:07:50.705 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313219 for osd.6 2024-04-26T01:07:50.706 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.744 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.5 2024-04-26T01:07:50.744 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.751 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.11 2024-04-26T01:07:50.751 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.761 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313219 for osd.8 2024-04-26T01:07:50.761 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.765 INFO:teuthology.orchestra.run.smithi008.stdout:90194313220 2024-04-26T01:07:50.787 INFO:teuthology.orchestra.run.smithi008.stdout:90194313219 2024-04-26T01:07:50.793 INFO:teuthology.orchestra.run.smithi008.stdout:90194313221 2024-04-26T01:07:50.803 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.9 2024-04-26T01:07:50.803 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.816 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313219 for osd.0 2024-04-26T01:07:50.816 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.838 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313221 for osd.15 2024-04-26T01:07:50.838 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.851 INFO:teuthology.orchestra.run.smithi008.stdout:94489280516 2024-04-26T01:07:50.855 INFO:teuthology.orchestra.run.smithi008.stdout:90194313220 2024-04-26T01:07:50.870 INFO:teuthology.orchestra.run.smithi008.stdout:94489280514 2024-04-26T01:07:50.873 INFO:teuthology.orchestra.run.smithi008.stdout:90194313219 2024-04-26T01:07:50.874 INFO:teuthology.orchestra.run.smithi008.stdout:90194313220 2024-04-26T01:07:50.885 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.7 2024-04-26T01:07:50.885 INFO:teuthology.orchestra.run.smithi008.stdout:90194313220 2024-04-26T01:07:50.885 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.887 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.1 2024-04-26T01:07:50.887 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.899 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313219 for osd.10 2024-04-26T01:07:50.900 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280514 got 94489280514 for osd.13 2024-04-26T01:07:50.912 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.914 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.2 2024-04-26T01:07:50.914 INFO:teuthology.orchestra.run.smithi008.stdout:90194313221 2024-04-26T01:07:50.914 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.918 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.14 2024-04-26T01:07:50.918 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.920 INFO:teuthology.orchestra.run.smithi008.stdout:94489280515 2024-04-26T01:07:50.930 INFO:teuthology.orchestra.run.smithi008.stdout:90194313219 2024-04-26T01:07:50.935 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313221 for osd.12 2024-04-26T01:07:50.935 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.943 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.3 2024-04-26T01:07:50.943 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.948 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313219 for osd.4 2024-04-26T01:07:50.948 DEBUG:teuthology.parallel:result is None 2024-04-26T01:07:50.949 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T01:07:50.949 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:07:51.276 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:07:51.276 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-26T01:07:51.293 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-04-26T01:07:49.982942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":189,"num_write_kb":3198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":444928,"kb_used_data":4160,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504512,"statfs":{"total":1546188226560,"available":1545732620288,"internally_reserved":0,"allocated":4259840,"data_stored":2712296,"data_compressed":240688,"data_compressed_allocated":385024,"data_compressed_original":1523712,"omap_allocated":25522,"internal_metadata":450993230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:07:48.153457+0000","last_change":"2024-04-26T01:07:48.149017+0000","last_active":"2024-04-26T01:07:48.153457+0000","last_peered":"2024-04-26T01:07:48.153457+0000","last_clean":"2024-04-26T01:07:48.153457+0000","last_became_active":"2024-04-26T01:07:48.148023+0000","last_became_peered":"2024-04-26T01:07:48.148023+0000","last_unstale":"2024-04-26T01:07:48.153457+0000","last_undegraded":"2024-04-26T01:07:48.153457+0000","last_fullsized":"2024-04-26T01:07:48.153457+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:57.724356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.277344+0000","last_change":"2024-04-26T01:07:48.145909+0000","last_active":"2024-04-26T01:07:49.277344+0000","last_peered":"2024-04-26T01:07:49.277344+0000","last_clean":"2024-04-26T01:07:49.277344+0000","last_became_active":"2024-04-26T01:07:48.145512+0000","last_became_peered":"2024-04-26T01:07:48.145512+0000","last_unstale":"2024-04-26T01:07:49.277344+0000","last_undegraded":"2024-04-26T01:07:49.277344+0000","last_fullsized":"2024-04-26T01:07:49.277344+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:16.344494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:07:48.146493+0000","last_change":"2024-04-26T01:07:48.145645+0000","last_active":"2024-04-26T01:07:48.146493+0000","last_peered":"2024-04-26T01:07:48.146493+0000","last_clean":"2024-04-26T01:07:48.146493+0000","last_became_active":"2024-04-26T01:07:48.145376+0000","last_became_peered":"2024-04-26T01:07:48.145376+0000","last_unstale":"2024-04-26T01:07:48.146493+0000","last_undegraded":"2024-04-26T01:07:48.146493+0000","last_fullsized":"2024-04-26T01:07:48.146493+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:15.520460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.277167+0000","last_change":"2024-04-26T01:07:48.465954+0000","last_active":"2024-04-26T01:07:49.277167+0000","last_peered":"2024-04-26T01:07:49.277167+0000","last_clean":"2024-04-26T01:07:49.277167+0000","last_became_active":"2024-04-26T01:07:48.465232+0000","last_became_peered":"2024-04-26T01:07:48.465232+0000","last_unstale":"2024-04-26T01:07:49.277167+0000","last_undegraded":"2024-04-26T01:07:49.277167+0000","last_fullsized":"2024-04-26T01:07:49.277167+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:49.270720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.179987+0000","last_change":"2024-04-26T01:07:48.147756+0000","last_active":"2024-04-26T01:07:49.179987+0000","last_peered":"2024-04-26T01:07:49.179987+0000","last_clean":"2024-04-26T01:07:49.179987+0000","last_became_active":"2024-04-26T01:07:48.147003+0000","last_became_peered":"2024-04-26T01:07:48.147003+0000","last_unstale":"2024-04-26T01:07:49.179987+0000","last_undegraded":"2024-04-26T01:07:49.179987+0000","last_fullsized":"2024-04-26T01:07:49.179987+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:05.491411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.139796+0000","last_change":"2024-04-26T01:07:48.146870+0000","last_active":"2024-04-26T01:07:49.139796+0000","last_peered":"2024-04-26T01:07:49.139796+0000","last_clean":"2024-04-26T01:07:49.139796+0000","last_became_active":"2024-04-26T01:07:48.145991+0000","last_became_peered":"2024-04-26T01:07:48.145991+0000","last_unstale":"2024-04-26T01:07:49.139796+0000","last_undegraded":"2024-04-26T01:07:49.139796+0000","last_fullsized":"2024-04-26T01:07:49.139796+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:13.523679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.180018+0000","last_change":"2024-04-26T01:07:48.149275+0000","last_active":"2024-04-26T01:07:49.180018+0000","last_peered":"2024-04-26T01:07:49.180018+0000","last_clean":"2024-04-26T01:07:49.180018+0000","last_became_active":"2024-04-26T01:07:48.148395+0000","last_became_peered":"2024-04-26T01:07:48.148395+0000","last_unstale":"2024-04-26T01:07:49.180018+0000","last_undegraded":"2024-04-26T01:07:49.180018+0000","last_fullsized":"2024-04-26T01:07:49.180018+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:49.139938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.496480+0000","last_change":"2024-04-26T01:07:48.145129+0000","last_active":"2024-04-26T01:07:49.496480+0000","last_peered":"2024-04-26T01:07:49.496480+0000","last_clean":"2024-04-26T01:07:49.496480+0000","last_became_active":"2024-04-26T01:07:48.144313+0000","last_became_peered":"2024-04-26T01:07:48.144313+0000","last_unstale":"2024-04-26T01:07:49.496480+0000","last_undegraded":"2024-04-26T01:07:49.496480+0000","last_fullsized":"2024-04-26T01:07:49.496480+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:03.023694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"26'145","reported_seq":197,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.139770+0000","last_change":"2024-04-26T01:07:46.409257+0000","last_active":"2024-04-26T01:07:49.139770+0000","last_peered":"2024-04-26T01:07:49.139770+0000","last_clean":"2024-04-26T01:07:49.139770+0000","last_became_active":"2024-04-26T01:07:46.407301+0000","last_became_peered":"2024-04-26T01:07:46.407301+0000","last_unstale":"2024-04-26T01:07:49.139770+0000","last_undegraded":"2024-04-26T01:07:49.139770+0000","last_fullsized":"2024-04-26T01:07:49.139770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_clean_scrub_stamp":"2024-04-26T01:07:45.031420+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.371437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":189,"num_write_kb":3198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":189,"num_write_kb":3198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":934976,"data_compressed":22544,"data_compressed_allocated":73728,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103986,"data_compressed":11465,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280514,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":319488,"data_stored":580936,"data_compressed":25629,"data_compressed_allocated":57344,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313219,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":249856,"data_stored":103986,"data_compressed":11465,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":319488,"data_stored":580936,"data_compressed":25629,"data_compressed_allocated":57344,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":103986,"data_compressed":11465,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,6,7,8,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":103986,"data_compressed":11465,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,4,5,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":21,"seq":90194313219,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,4,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,4,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,5,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":11272,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":11272,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:07:51.294 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:07:51.587 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:07:51.587 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-26T01:07:51.603 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-04-26T01:07:49.982942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":189,"num_write_kb":3198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":444928,"kb_used_data":4160,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504512,"statfs":{"total":1546188226560,"available":1545732620288,"internally_reserved":0,"allocated":4259840,"data_stored":2712296,"data_compressed":240688,"data_compressed_allocated":385024,"data_compressed_original":1523712,"omap_allocated":25522,"internal_metadata":450993230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:07:48.153457+0000","last_change":"2024-04-26T01:07:48.149017+0000","last_active":"2024-04-26T01:07:48.153457+0000","last_peered":"2024-04-26T01:07:48.153457+0000","last_clean":"2024-04-26T01:07:48.153457+0000","last_became_active":"2024-04-26T01:07:48.148023+0000","last_became_peered":"2024-04-26T01:07:48.148023+0000","last_unstale":"2024-04-26T01:07:48.153457+0000","last_undegraded":"2024-04-26T01:07:48.153457+0000","last_fullsized":"2024-04-26T01:07:48.153457+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:57.724356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.277344+0000","last_change":"2024-04-26T01:07:48.145909+0000","last_active":"2024-04-26T01:07:49.277344+0000","last_peered":"2024-04-26T01:07:49.277344+0000","last_clean":"2024-04-26T01:07:49.277344+0000","last_became_active":"2024-04-26T01:07:48.145512+0000","last_became_peered":"2024-04-26T01:07:48.145512+0000","last_unstale":"2024-04-26T01:07:49.277344+0000","last_undegraded":"2024-04-26T01:07:49.277344+0000","last_fullsized":"2024-04-26T01:07:49.277344+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:16.344494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:07:48.146493+0000","last_change":"2024-04-26T01:07:48.145645+0000","last_active":"2024-04-26T01:07:48.146493+0000","last_peered":"2024-04-26T01:07:48.146493+0000","last_clean":"2024-04-26T01:07:48.146493+0000","last_became_active":"2024-04-26T01:07:48.145376+0000","last_became_peered":"2024-04-26T01:07:48.145376+0000","last_unstale":"2024-04-26T01:07:48.146493+0000","last_undegraded":"2024-04-26T01:07:48.146493+0000","last_fullsized":"2024-04-26T01:07:48.146493+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:15.520460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.277167+0000","last_change":"2024-04-26T01:07:48.465954+0000","last_active":"2024-04-26T01:07:49.277167+0000","last_peered":"2024-04-26T01:07:49.277167+0000","last_clean":"2024-04-26T01:07:49.277167+0000","last_became_active":"2024-04-26T01:07:48.465232+0000","last_became_peered":"2024-04-26T01:07:48.465232+0000","last_unstale":"2024-04-26T01:07:49.277167+0000","last_undegraded":"2024-04-26T01:07:49.277167+0000","last_fullsized":"2024-04-26T01:07:49.277167+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:49.270720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.179987+0000","last_change":"2024-04-26T01:07:48.147756+0000","last_active":"2024-04-26T01:07:49.179987+0000","last_peered":"2024-04-26T01:07:49.179987+0000","last_clean":"2024-04-26T01:07:49.179987+0000","last_became_active":"2024-04-26T01:07:48.147003+0000","last_became_peered":"2024-04-26T01:07:48.147003+0000","last_unstale":"2024-04-26T01:07:49.179987+0000","last_undegraded":"2024-04-26T01:07:49.179987+0000","last_fullsized":"2024-04-26T01:07:49.179987+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:05.491411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.139796+0000","last_change":"2024-04-26T01:07:48.146870+0000","last_active":"2024-04-26T01:07:49.139796+0000","last_peered":"2024-04-26T01:07:49.139796+0000","last_clean":"2024-04-26T01:07:49.139796+0000","last_became_active":"2024-04-26T01:07:48.145991+0000","last_became_peered":"2024-04-26T01:07:48.145991+0000","last_unstale":"2024-04-26T01:07:49.139796+0000","last_undegraded":"2024-04-26T01:07:49.139796+0000","last_fullsized":"2024-04-26T01:07:49.139796+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:13.523679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.180018+0000","last_change":"2024-04-26T01:07:48.149275+0000","last_active":"2024-04-26T01:07:49.180018+0000","last_peered":"2024-04-26T01:07:49.180018+0000","last_clean":"2024-04-26T01:07:49.180018+0000","last_became_active":"2024-04-26T01:07:48.148395+0000","last_became_peered":"2024-04-26T01:07:48.148395+0000","last_unstale":"2024-04-26T01:07:49.180018+0000","last_undegraded":"2024-04-26T01:07:49.180018+0000","last_fullsized":"2024-04-26T01:07:49.180018+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:49.139938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.496480+0000","last_change":"2024-04-26T01:07:48.145129+0000","last_active":"2024-04-26T01:07:49.496480+0000","last_peered":"2024-04-26T01:07:49.496480+0000","last_clean":"2024-04-26T01:07:49.496480+0000","last_became_active":"2024-04-26T01:07:48.144313+0000","last_became_peered":"2024-04-26T01:07:48.144313+0000","last_unstale":"2024-04-26T01:07:49.496480+0000","last_undegraded":"2024-04-26T01:07:49.496480+0000","last_fullsized":"2024-04-26T01:07:49.496480+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:03.023694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"26'145","reported_seq":197,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.139770+0000","last_change":"2024-04-26T01:07:46.409257+0000","last_active":"2024-04-26T01:07:49.139770+0000","last_peered":"2024-04-26T01:07:49.139770+0000","last_clean":"2024-04-26T01:07:49.139770+0000","last_became_active":"2024-04-26T01:07:46.407301+0000","last_became_peered":"2024-04-26T01:07:46.407301+0000","last_unstale":"2024-04-26T01:07:49.139770+0000","last_undegraded":"2024-04-26T01:07:49.139770+0000","last_fullsized":"2024-04-26T01:07:49.139770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_clean_scrub_stamp":"2024-04-26T01:07:45.031420+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.371437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":189,"num_write_kb":3198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":189,"num_write_kb":3198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":934976,"data_compressed":22544,"data_compressed_allocated":73728,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103986,"data_compressed":11465,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280514,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":319488,"data_stored":580936,"data_compressed":25629,"data_compressed_allocated":57344,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313219,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":249856,"data_stored":103986,"data_compressed":11465,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":319488,"data_stored":580936,"data_compressed":25629,"data_compressed_allocated":57344,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":103986,"data_compressed":11465,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,6,7,8,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":103986,"data_compressed":11465,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,4,5,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":21,"seq":90194313219,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,4,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,4,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,5,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":11272,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":11272,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:07:51.604 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T01:07:51.604 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T01:07:51.604 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-26T01:07:51.604 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-26T01:07:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:07:51.924 INFO:teuthology.orchestra.run.smithi008.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T01:07:51.942 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-26T01:07:51.943 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-26T01:07:51.955 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-26T01:07:51.955 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'}} 2024-04-26T01:07:51.955 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-04-26T01:07:51.955 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-26T01:07:52.271 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-26T01:07:52.271 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T01:07:52.271 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:07:52.568 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:07:52.569 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-26T01:07:52.585 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-26T01:07:51.983414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":189,"num_write_kb":3198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":444944,"kb_used_data":4172,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504496,"statfs":{"total":1546188226560,"available":1545732603904,"internally_reserved":0,"allocated":4272128,"data_stored":2721758,"data_compressed":243580,"data_compressed_allocated":389120,"data_compressed_original":1531904,"omap_allocated":25522,"internal_metadata":450993230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000471"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:07:48.153457+0000","last_change":"2024-04-26T01:07:48.149017+0000","last_active":"2024-04-26T01:07:48.153457+0000","last_peered":"2024-04-26T01:07:48.153457+0000","last_clean":"2024-04-26T01:07:48.153457+0000","last_became_active":"2024-04-26T01:07:48.148023+0000","last_became_peered":"2024-04-26T01:07:48.148023+0000","last_unstale":"2024-04-26T01:07:48.153457+0000","last_undegraded":"2024-04-26T01:07:48.153457+0000","last_fullsized":"2024-04-26T01:07:48.153457+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:57.724356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.277344+0000","last_change":"2024-04-26T01:07:48.145909+0000","last_active":"2024-04-26T01:07:49.277344+0000","last_peered":"2024-04-26T01:07:49.277344+0000","last_clean":"2024-04-26T01:07:49.277344+0000","last_became_active":"2024-04-26T01:07:48.145512+0000","last_became_peered":"2024-04-26T01:07:48.145512+0000","last_unstale":"2024-04-26T01:07:49.277344+0000","last_undegraded":"2024-04-26T01:07:49.277344+0000","last_fullsized":"2024-04-26T01:07:49.277344+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:16.344494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:07:48.146493+0000","last_change":"2024-04-26T01:07:48.145645+0000","last_active":"2024-04-26T01:07:48.146493+0000","last_peered":"2024-04-26T01:07:48.146493+0000","last_clean":"2024-04-26T01:07:48.146493+0000","last_became_active":"2024-04-26T01:07:48.145376+0000","last_became_peered":"2024-04-26T01:07:48.145376+0000","last_unstale":"2024-04-26T01:07:48.146493+0000","last_undegraded":"2024-04-26T01:07:48.146493+0000","last_fullsized":"2024-04-26T01:07:48.146493+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:15.520460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.277167+0000","last_change":"2024-04-26T01:07:48.465954+0000","last_active":"2024-04-26T01:07:49.277167+0000","last_peered":"2024-04-26T01:07:49.277167+0000","last_clean":"2024-04-26T01:07:49.277167+0000","last_became_active":"2024-04-26T01:07:48.465232+0000","last_became_peered":"2024-04-26T01:07:48.465232+0000","last_unstale":"2024-04-26T01:07:49.277167+0000","last_undegraded":"2024-04-26T01:07:49.277167+0000","last_fullsized":"2024-04-26T01:07:49.277167+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:49.270720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.179987+0000","last_change":"2024-04-26T01:07:48.147756+0000","last_active":"2024-04-26T01:07:49.179987+0000","last_peered":"2024-04-26T01:07:49.179987+0000","last_clean":"2024-04-26T01:07:49.179987+0000","last_became_active":"2024-04-26T01:07:48.147003+0000","last_became_peered":"2024-04-26T01:07:48.147003+0000","last_unstale":"2024-04-26T01:07:49.179987+0000","last_undegraded":"2024-04-26T01:07:49.179987+0000","last_fullsized":"2024-04-26T01:07:49.179987+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:05.491411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.139796+0000","last_change":"2024-04-26T01:07:48.146870+0000","last_active":"2024-04-26T01:07:49.139796+0000","last_peered":"2024-04-26T01:07:49.139796+0000","last_clean":"2024-04-26T01:07:49.139796+0000","last_became_active":"2024-04-26T01:07:48.145991+0000","last_became_peered":"2024-04-26T01:07:48.145991+0000","last_unstale":"2024-04-26T01:07:49.139796+0000","last_undegraded":"2024-04-26T01:07:49.139796+0000","last_fullsized":"2024-04-26T01:07:49.139796+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:13.523679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.180018+0000","last_change":"2024-04-26T01:07:48.149275+0000","last_active":"2024-04-26T01:07:49.180018+0000","last_peered":"2024-04-26T01:07:49.180018+0000","last_clean":"2024-04-26T01:07:49.180018+0000","last_became_active":"2024-04-26T01:07:48.148395+0000","last_became_peered":"2024-04-26T01:07:48.148395+0000","last_unstale":"2024-04-26T01:07:49.180018+0000","last_undegraded":"2024-04-26T01:07:49.180018+0000","last_fullsized":"2024-04-26T01:07:49.180018+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:49.139938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.496480+0000","last_change":"2024-04-26T01:07:48.145129+0000","last_active":"2024-04-26T01:07:49.496480+0000","last_peered":"2024-04-26T01:07:49.496480+0000","last_clean":"2024-04-26T01:07:49.496480+0000","last_became_active":"2024-04-26T01:07:48.144313+0000","last_became_peered":"2024-04-26T01:07:48.144313+0000","last_unstale":"2024-04-26T01:07:49.496480+0000","last_undegraded":"2024-04-26T01:07:49.496480+0000","last_fullsized":"2024-04-26T01:07:49.496480+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:03.023694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"26'145","reported_seq":197,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.139770+0000","last_change":"2024-04-26T01:07:46.409257+0000","last_active":"2024-04-26T01:07:49.139770+0000","last_peered":"2024-04-26T01:07:49.139770+0000","last_clean":"2024-04-26T01:07:49.139770+0000","last_became_active":"2024-04-26T01:07:46.407301+0000","last_became_peered":"2024-04-26T01:07:46.407301+0000","last_unstale":"2024-04-26T01:07:49.139770+0000","last_undegraded":"2024-04-26T01:07:49.139770+0000","last_fullsized":"2024-04-26T01:07:49.139770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_clean_scrub_stamp":"2024-04-26T01:07:45.031420+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.371437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":189,"num_write_kb":3198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":189,"num_write_kb":3198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":934976,"data_compressed":22544,"data_compressed_allocated":73728,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280515,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":319488,"data_stored":580936,"data_compressed":25629,"data_compressed_allocated":57344,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313219,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":249856,"data_stored":103986,"data_compressed":11465,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":319488,"data_stored":580936,"data_compressed":25629,"data_compressed_allocated":57344,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":103986,"data_compressed":11465,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,6,7,8,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":103986,"data_compressed":11465,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,4,5,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":21,"seq":90194313219,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,4,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,4,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,5,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":11272,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":11272,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:07:52.586 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:07:52.885 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:07:52.885 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-26T01:07:52.903 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-26T01:07:51.983414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":189,"num_write_kb":3198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":444944,"kb_used_data":4172,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504496,"statfs":{"total":1546188226560,"available":1545732603904,"internally_reserved":0,"allocated":4272128,"data_stored":2721758,"data_compressed":243580,"data_compressed_allocated":389120,"data_compressed_original":1531904,"omap_allocated":25522,"internal_metadata":450993230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000471"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:07:48.153457+0000","last_change":"2024-04-26T01:07:48.149017+0000","last_active":"2024-04-26T01:07:48.153457+0000","last_peered":"2024-04-26T01:07:48.153457+0000","last_clean":"2024-04-26T01:07:48.153457+0000","last_became_active":"2024-04-26T01:07:48.148023+0000","last_became_peered":"2024-04-26T01:07:48.148023+0000","last_unstale":"2024-04-26T01:07:48.153457+0000","last_undegraded":"2024-04-26T01:07:48.153457+0000","last_fullsized":"2024-04-26T01:07:48.153457+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:57.724356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.277344+0000","last_change":"2024-04-26T01:07:48.145909+0000","last_active":"2024-04-26T01:07:49.277344+0000","last_peered":"2024-04-26T01:07:49.277344+0000","last_clean":"2024-04-26T01:07:49.277344+0000","last_became_active":"2024-04-26T01:07:48.145512+0000","last_became_peered":"2024-04-26T01:07:48.145512+0000","last_unstale":"2024-04-26T01:07:49.277344+0000","last_undegraded":"2024-04-26T01:07:49.277344+0000","last_fullsized":"2024-04-26T01:07:49.277344+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:16.344494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:07:48.146493+0000","last_change":"2024-04-26T01:07:48.145645+0000","last_active":"2024-04-26T01:07:48.146493+0000","last_peered":"2024-04-26T01:07:48.146493+0000","last_clean":"2024-04-26T01:07:48.146493+0000","last_became_active":"2024-04-26T01:07:48.145376+0000","last_became_peered":"2024-04-26T01:07:48.145376+0000","last_unstale":"2024-04-26T01:07:48.146493+0000","last_undegraded":"2024-04-26T01:07:48.146493+0000","last_fullsized":"2024-04-26T01:07:48.146493+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:15.520460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.277167+0000","last_change":"2024-04-26T01:07:48.465954+0000","last_active":"2024-04-26T01:07:49.277167+0000","last_peered":"2024-04-26T01:07:49.277167+0000","last_clean":"2024-04-26T01:07:49.277167+0000","last_became_active":"2024-04-26T01:07:48.465232+0000","last_became_peered":"2024-04-26T01:07:48.465232+0000","last_unstale":"2024-04-26T01:07:49.277167+0000","last_undegraded":"2024-04-26T01:07:49.277167+0000","last_fullsized":"2024-04-26T01:07:49.277167+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:49.270720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.179987+0000","last_change":"2024-04-26T01:07:48.147756+0000","last_active":"2024-04-26T01:07:49.179987+0000","last_peered":"2024-04-26T01:07:49.179987+0000","last_clean":"2024-04-26T01:07:49.179987+0000","last_became_active":"2024-04-26T01:07:48.147003+0000","last_became_peered":"2024-04-26T01:07:48.147003+0000","last_unstale":"2024-04-26T01:07:49.179987+0000","last_undegraded":"2024-04-26T01:07:49.179987+0000","last_fullsized":"2024-04-26T01:07:49.179987+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:05.491411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.139796+0000","last_change":"2024-04-26T01:07:48.146870+0000","last_active":"2024-04-26T01:07:49.139796+0000","last_peered":"2024-04-26T01:07:49.139796+0000","last_clean":"2024-04-26T01:07:49.139796+0000","last_became_active":"2024-04-26T01:07:48.145991+0000","last_became_peered":"2024-04-26T01:07:48.145991+0000","last_unstale":"2024-04-26T01:07:49.139796+0000","last_undegraded":"2024-04-26T01:07:49.139796+0000","last_fullsized":"2024-04-26T01:07:49.139796+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:13.523679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.180018+0000","last_change":"2024-04-26T01:07:48.149275+0000","last_active":"2024-04-26T01:07:49.180018+0000","last_peered":"2024-04-26T01:07:49.180018+0000","last_clean":"2024-04-26T01:07:49.180018+0000","last_became_active":"2024-04-26T01:07:48.148395+0000","last_became_peered":"2024-04-26T01:07:48.148395+0000","last_unstale":"2024-04-26T01:07:49.180018+0000","last_undegraded":"2024-04-26T01:07:49.180018+0000","last_fullsized":"2024-04-26T01:07:49.180018+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:49.139938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.496480+0000","last_change":"2024-04-26T01:07:48.145129+0000","last_active":"2024-04-26T01:07:49.496480+0000","last_peered":"2024-04-26T01:07:49.496480+0000","last_clean":"2024-04-26T01:07:49.496480+0000","last_became_active":"2024-04-26T01:07:48.144313+0000","last_became_peered":"2024-04-26T01:07:48.144313+0000","last_unstale":"2024-04-26T01:07:49.496480+0000","last_undegraded":"2024-04-26T01:07:49.496480+0000","last_fullsized":"2024-04-26T01:07:49.496480+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:03.023694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"26'145","reported_seq":197,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.139770+0000","last_change":"2024-04-26T01:07:46.409257+0000","last_active":"2024-04-26T01:07:49.139770+0000","last_peered":"2024-04-26T01:07:49.139770+0000","last_clean":"2024-04-26T01:07:49.139770+0000","last_became_active":"2024-04-26T01:07:46.407301+0000","last_became_peered":"2024-04-26T01:07:46.407301+0000","last_unstale":"2024-04-26T01:07:49.139770+0000","last_undegraded":"2024-04-26T01:07:49.139770+0000","last_fullsized":"2024-04-26T01:07:49.139770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_clean_scrub_stamp":"2024-04-26T01:07:45.031420+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.371437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":189,"num_write_kb":3198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":189,"num_write_kb":3198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":934976,"data_compressed":22544,"data_compressed_allocated":73728,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280515,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":319488,"data_stored":580936,"data_compressed":25629,"data_compressed_allocated":57344,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313219,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":249856,"data_stored":103986,"data_compressed":11465,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":319488,"data_stored":580936,"data_compressed":25629,"data_compressed_allocated":57344,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":103986,"data_compressed":11465,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,6,7,8,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":103986,"data_compressed":11465,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,4,5,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":21,"seq":90194313219,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,4,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,4,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,5,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":11272,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":11272,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:07:52.904 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T01:07:52.904 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-26T01:07:53.217 INFO:teuthology.orchestra.run.smithi008.stdout:epoch 27 2024-04-26T01:07:53.217 INFO:teuthology.orchestra.run.smithi008.stdout:fsid 4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554 2024-04-26T01:07:53.217 INFO:teuthology.orchestra.run.smithi008.stdout:created 2024-04-26T01:07:31.350368+0000 2024-04-26T01:07:53.217 INFO:teuthology.orchestra.run.smithi008.stdout:modified 2024-04-26T01:07:49.132650+0000 2024-04-26T01:07:53.217 INFO:teuthology.orchestra.run.smithi008.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-26T01:07:53.217 INFO:teuthology.orchestra.run.smithi008.stdout:crush_version 4 2024-04-26T01:07:53.217 INFO:teuthology.orchestra.run.smithi008.stdout:full_ratio 0.95 2024-04-26T01:07:53.217 INFO:teuthology.orchestra.run.smithi008.stdout:backfillfull_ratio 0.9 2024-04-26T01:07:53.217 INFO:teuthology.orchestra.run.smithi008.stdout:nearfull_ratio 0.85 2024-04-26T01:07:53.217 INFO:teuthology.orchestra.run.smithi008.stdout:require_min_compat_client luminous 2024-04-26T01:07:53.218 INFO:teuthology.orchestra.run.smithi008.stdout:min_compat_client jewel 2024-04-26T01:07:53.218 INFO:teuthology.orchestra.run.smithi008.stdout:require_osd_release squid 2024-04-26T01:07:53.218 INFO:teuthology.orchestra.run.smithi008.stdout:stretch_mode_enabled false 2024-04-26T01:07:53.218 INFO:teuthology.orchestra.run.smithi008.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2024-04-26T01:07:53.218 INFO:teuthology.orchestra.run.smithi008.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 27 flags hashpspool stripe_width 0 application rbd read_balance_score 4.02 2024-04-26T01:07:53.218 INFO:teuthology.orchestra.run.smithi008.stdout:max_osd 16 2024-04-26T01:07:53.218 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6800/188308042,v1:172.21.15.8:6801/188308042] [v2:172.21.15.8:6802/188308042,v1:172.21.15.8:6803/188308042] exists,up abed240a-e896-459b-8a43-cd1419cafbc2 2024-04-26T01:07:53.218 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6816/3487767810,v1:172.21.15.12:6819/3487767810] [v2:172.21.15.12:6821/3487767810,v1:172.21.15.12:6823/3487767810] exists,up 831c717c-e124-4489-a44f-93db78c47deb 2024-04-26T01:07:53.218 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6803/3599623558,v1:172.21.15.116:6805/3599623558] [v2:172.21.15.116:6807/3599623558,v1:172.21.15.116:6809/3599623558] exists,up 92a95165-ae34-4f29-a7f3-fe97afc26af7 2024-04-26T01:07:53.218 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6802/4123918458,v1:172.21.15.164:6804/4123918458] [v2:172.21.15.164:6806/4123918458,v1:172.21.15.164:6808/4123918458] exists,up e9d8f764-4951-4de4-92c7-682850852e85 2024-04-26T01:07:53.219 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6824/425515720,v1:172.21.15.8:6825/425515720] [v2:172.21.15.8:6826/425515720,v1:172.21.15.8:6827/425515720] exists,up 7f5e93f8-b1f6-41e7-8da9-b8e57d4259bc 2024-04-26T01:07:53.219 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6808/1641884791,v1:172.21.15.12:6809/1641884791] [v2:172.21.15.12:6810/1641884791,v1:172.21.15.12:6811/1641884791] exists,up f7b3a311-3bf6-4ae5-b7d2-dedbbe6c47ea 2024-04-26T01:07:53.219 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6816/156763037,v1:172.21.15.116:6817/156763037] [v2:172.21.15.116:6819/156763037,v1:172.21.15.116:6820/156763037] exists,up 796d7212-13de-43b2-8aa4-4abc4fd210a0 2024-04-26T01:07:53.219 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6824/2291946233,v1:172.21.15.164:6826/2291946233] [v2:172.21.15.164:6828/2291946233,v1:172.21.15.164:6829/2291946233] exists,up ccb8cd0f-a0d6-44d7-86cc-809eff9687ce 2024-04-26T01:07:53.219 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6809/2709504491,v1:172.21.15.8:6811/2709504491] [v2:172.21.15.8:6813/2709504491,v1:172.21.15.8:6815/2709504491] exists,up bf13e311-71b7-44ff-af1b-68d118935c39 2024-04-26T01:07:53.219 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6800/1155130892,v1:172.21.15.12:6801/1155130892] [v2:172.21.15.12:6802/1155130892,v1:172.21.15.12:6803/1155130892] exists,up 4864f46f-fd7b-48fc-908b-2564f7bda67b 2024-04-26T01:07:53.219 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6818/1461831728,v1:172.21.15.116:6822/1461831728] [v2:172.21.15.116:6826/1461831728,v1:172.21.15.116:6827/1461831728] exists,up 09cbb9ca-f609-409e-821b-93e695b87067 2024-04-26T01:07:53.219 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6818/1282808378,v1:172.21.15.164:6819/1282808378] [v2:172.21.15.164:6820/1282808378,v1:172.21.15.164:6821/1282808378] exists,up 9bfdf353-5174-40dd-bb3c-ab2dc850adad 2024-04-26T01:07:53.219 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6808/2082773890,v1:172.21.15.8:6810/2082773890] [v2:172.21.15.8:6812/2082773890,v1:172.21.15.8:6814/2082773890] exists,up a3e244d8-a765-467e-820c-caa5d1d97c9f 2024-04-26T01:07:53.219 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6817/3158583580,v1:172.21.15.12:6818/3158583580] [v2:172.21.15.12:6820/3158583580,v1:172.21.15.12:6822/3158583580] exists,up 383875c2-0a0c-4834-8beb-a6784eef1f9f 2024-04-26T01:07:53.219 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6800/2158650188,v1:172.21.15.116:6801/2158650188] [v2:172.21.15.116:6802/2158650188,v1:172.21.15.116:6804/2158650188] exists,up 30311d57-c625-4dff-98d5-1ba0a84c0ceb 2024-04-26T01:07:53.219 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] [v2:172.21.15.164:6807/2601606699,v1:172.21.15.164:6809/2601606699] exists,up 1243478f-9908-460d-88d8-3c542d5258b0 2024-04-26T01:07:53.235 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6800/188308042,v1:172.21.15.8:6801/188308042] [v2:172.21.15.8:6802/188308042,v1:172.21.15.8:6803/188308042] exists,up abed240a-e896-459b-8a43-cd1419cafbc2', 'osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6816/3487767810,v1:172.21.15.12:6819/3487767810] [v2:172.21.15.12:6821/3487767810,v1:172.21.15.12:6823/3487767810] exists,up 831c717c-e124-4489-a44f-93db78c47deb', 'osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6803/3599623558,v1:172.21.15.116:6805/3599623558] [v2:172.21.15.116:6807/3599623558,v1:172.21.15.116:6809/3599623558] exists,up 92a95165-ae34-4f29-a7f3-fe97afc26af7', 'osd.3 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6802/4123918458,v1:172.21.15.164:6804/4123918458] [v2:172.21.15.164:6806/4123918458,v1:172.21.15.164:6808/4123918458] exists,up e9d8f764-4951-4de4-92c7-682850852e85', 'osd.4 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6824/425515720,v1:172.21.15.8:6825/425515720] [v2:172.21.15.8:6826/425515720,v1:172.21.15.8:6827/425515720] exists,up 7f5e93f8-b1f6-41e7-8da9-b8e57d4259bc', 'osd.5 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6808/1641884791,v1:172.21.15.12:6809/1641884791] [v2:172.21.15.12:6810/1641884791,v1:172.21.15.12:6811/1641884791] exists,up f7b3a311-3bf6-4ae5-b7d2-dedbbe6c47ea', 'osd.6 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6816/156763037,v1:172.21.15.116:6817/156763037] [v2:172.21.15.116:6819/156763037,v1:172.21.15.116:6820/156763037] exists,up 796d7212-13de-43b2-8aa4-4abc4fd210a0', 'osd.7 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6824/2291946233,v1:172.21.15.164:6826/2291946233] [v2:172.21.15.164:6828/2291946233,v1:172.21.15.164:6829/2291946233] exists,up ccb8cd0f-a0d6-44d7-86cc-809eff9687ce', 'osd.8 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6809/2709504491,v1:172.21.15.8:6811/2709504491] [v2:172.21.15.8:6813/2709504491,v1:172.21.15.8:6815/2709504491] exists,up bf13e311-71b7-44ff-af1b-68d118935c39', 'osd.9 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6800/1155130892,v1:172.21.15.12:6801/1155130892] [v2:172.21.15.12:6802/1155130892,v1:172.21.15.12:6803/1155130892] exists,up 4864f46f-fd7b-48fc-908b-2564f7bda67b', 'osd.10 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6818/1461831728,v1:172.21.15.116:6822/1461831728] [v2:172.21.15.116:6826/1461831728,v1:172.21.15.116:6827/1461831728] exists,up 09cbb9ca-f609-409e-821b-93e695b87067', 'osd.11 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6818/1282808378,v1:172.21.15.164:6819/1282808378] [v2:172.21.15.164:6820/1282808378,v1:172.21.15.164:6821/1282808378] exists,up 9bfdf353-5174-40dd-bb3c-ab2dc850adad', 'osd.12 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6808/2082773890,v1:172.21.15.8:6810/2082773890] [v2:172.21.15.8:6812/2082773890,v1:172.21.15.8:6814/2082773890] exists,up a3e244d8-a765-467e-820c-caa5d1d97c9f', 'osd.13 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6817/3158583580,v1:172.21.15.12:6818/3158583580] [v2:172.21.15.12:6820/3158583580,v1:172.21.15.12:6822/3158583580] exists,up 383875c2-0a0c-4834-8beb-a6784eef1f9f', 'osd.14 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6800/2158650188,v1:172.21.15.116:6801/2158650188] [v2:172.21.15.116:6802/2158650188,v1:172.21.15.116:6804/2158650188] exists,up 30311d57-c625-4dff-98d5-1ba0a84c0ceb', 'osd.15 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] [v2:172.21.15.164:6807/2601606699,v1:172.21.15.164:6809/2601606699] exists,up 1243478f-9908-460d-88d8-3c542d5258b0'] 2024-04-26T01:07:53.235 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-26T01:07:53.353 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:07:53.353 INFO:teuthology.orchestra.run.smithi008.stdout: "name": "mon.a", 2024-04-26T01:07:53.353 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster": "ceph", 2024-04-26T01:07:53.353 INFO:teuthology.orchestra.run.smithi008.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-26T01:07:53.353 INFO:teuthology.orchestra.run.smithi008.stdout: "admin_socket_mode": "", 2024-04-26T01:07:53.354 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-26T01:07:53.354 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_client_required": "cephx, none", 2024-04-26T01:07:53.354 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_cluster_required": "cephx", 2024-04-26T01:07:53.354 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_debug": "true", 2024-04-26T01:07:53.354 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-26T01:07:53.354 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-26T01:07:53.354 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_service_required": "cephx", 2024-04-26T01:07:53.354 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-26T01:07:53.354 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_supported": "cephx", 2024-04-26T01:07:53.354 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_aio": "true", 2024-04-26T01:07:53.354 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-26T01:07:53.354 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_aio_poll_ms": "250", 2024-04-26T01:07:53.355 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_aio_reap_max": "16", 2024-04-26T01:07:53.355 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_async_discard": "false", 2024-04-26T01:07:53.355 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_block_size": "4096", 2024-04-26T01:07:53.355 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_debug_aio": "false", 2024-04-26T01:07:53.355 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-26T01:07:53.355 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-26T01:07:53.355 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_debug_inflight_ios": "false", 2024-04-26T01:07:53.355 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_enable_discard": "false", 2024-04-26T01:07:53.355 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_flock_retry": "3", 2024-04-26T01:07:53.355 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-26T01:07:53.355 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_inject_crash": "0", 2024-04-26T01:07:53.355 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-26T01:07:53.355 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_ioring": "false", 2024-04-26T01:07:53.356 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_ioring_hipri": "false", 2024-04-26T01:07:53.356 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-26T01:07:53.356 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-26T01:07:53.356 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-26T01:07:53.356 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-26T01:07:53.356 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_type": "", 2024-04-26T01:07:53.356 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_alloc_size": "1048576", 2024-04-26T01:07:53.356 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_allocator": "hybrid", 2024-04-26T01:07:53.356 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_buffered_io": "true", 2024-04-26T01:07:53.356 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_check_for_zeros": "false", 2024-04-26T01:07:53.356 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-26T01:07:53.356 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-26T01:07:53.357 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_compact_log_sync": "false", 2024-04-26T01:07:53.357 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-26T01:07:53.357 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-26T01:07:53.357 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-26T01:07:53.357 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-26T01:07:53.357 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_max_log_runway": "4194304", 2024-04-26T01:07:53.357 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_max_prefetch": "1048576", 2024-04-26T01:07:53.357 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_min_flush_size": "524288", 2024-04-26T01:07:53.357 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_min_log_runway": "1048576", 2024-04-26T01:07:53.357 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_replay_recovery": "false", 2024-04-26T01:07:53.357 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-26T01:07:53.358 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-26T01:07:53.358 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_sync_write": "false", 2024-04-26T01:07:53.358 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-26T01:07:53.358 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-26T01:07:53.358 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-26T01:07:53.358 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_allocation_from_file": "true", 2024-04-26T01:07:53.358 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_allocator": "hybrid", 2024-04-26T01:07:53.358 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-26T01:07:53.358 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-26T01:07:53.358 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-26T01:07:53.358 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-26T01:07:53.358 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-26T01:07:53.358 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-26T01:07:53.359 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-26T01:07:53.359 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_create": "true", 2024-04-26T01:07:53.359 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_db_create": "false", 2024-04-26T01:07:53.359 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_db_path": "", 2024-04-26T01:07:53.359 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_db_size": "0", 2024-04-26T01:07:53.359 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_path": "", 2024-04-26T01:07:53.359 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_preallocate_file": "false", 2024-04-26T01:07:53.359 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_size": "107374182400", 2024-04-26T01:07:53.359 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_wal_create": "false", 2024-04-26T01:07:53.359 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_wal_path": "", 2024-04-26T01:07:53.359 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_wal_size": "100663296", 2024-04-26T01:07:53.359 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_bluefs": "true", 2024-04-26T01:07:53.359 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-26T01:07:53.359 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-26T01:07:53.359 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-26T01:07:53.360 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-26T01:07:53.360 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-26T01:07:53.360 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-26T01:07:53.360 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-26T01:07:53.360 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-26T01:07:53.360 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_autotune": "true", 2024-04-26T01:07:53.360 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-26T01:07:53.360 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-26T01:07:53.360 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-26T01:07:53.360 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-26T01:07:53.360 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_size": "0", 2024-04-26T01:07:53.360 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-26T01:07:53.360 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-26T01:07:53.360 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-26T01:07:53.360 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-26T01:07:53.361 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_type": "2q", 2024-04-26T01:07:53.361 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-26T01:07:53.361 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_clone_cow": "true", 2024-04-26T01:07:53.361 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-26T01:07:53.361 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-26T01:07:53.361 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-26T01:07:53.361 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-26T01:07:53.361 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-26T01:07:53.361 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-26T01:07:53.361 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-26T01:07:53.361 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_mode": "none", 2024-04-26T01:07:53.361 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-26T01:07:53.361 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_csum_type": "crc32c", 2024-04-26T01:07:53.361 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-26T01:07:53.361 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_freelist": "false", 2024-04-26T01:07:53.362 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-26T01:07:53.362 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-26T01:07:53.362 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-26T01:07:53.362 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-26T01:07:53.362 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-26T01:07:53.362 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-26T01:07:53.362 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-26T01:07:53.362 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-26T01:07:53.362 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-26T01:07:53.362 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-26T01:07:53.362 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-26T01:07:53.362 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-26T01:07:53.362 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-26T01:07:53.362 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_small_allocations": "0", 2024-04-26T01:07:53.363 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-26T01:07:53.363 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_default_buffered_read": "true", 2024-04-26T01:07:53.363 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_default_buffered_write": "false", 2024-04-26T01:07:53.363 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-26T01:07:53.363 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-26T01:07:53.363 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-26T01:07:53.363 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-26T01:07:53.363 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-26T01:07:53.363 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-26T01:07:53.364 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-26T01:07:53.364 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-26T01:07:53.364 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-26T01:07:53.364 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fail_eio": "false", 2024-04-26T01:07:53.364 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-26T01:07:53.364 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-26T01:07:53.364 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-26T01:07:53.364 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-26T01:07:53.364 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-26T01:07:53.364 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-26T01:07:53.364 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_on_mount": "false", 2024-04-26T01:07:53.364 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-26T01:07:53.365 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_on_umount": "false", 2024-04-26T01:07:53.365 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-26T01:07:53.365 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-26T01:07:53.365 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-26T01:07:53.365 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-26T01:07:53.365 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-26T01:07:53.365 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-26T01:07:53.365 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-26T01:07:53.366 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-26T01:07:53.366 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_ignore_data_csum": "false", 2024-04-26T01:07:53.366 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-26T01:07:53.366 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-26T01:07:53.366 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-26T01:07:53.366 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-26T01:07:53.366 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_log_op_age": "5.000000", 2024-04-26T01:07:53.366 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_max_alloc_size": "0", 2024-04-26T01:07:53.366 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_max_blob_size": "0", 2024-04-26T01:07:53.366 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-26T01:07:53.366 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-26T01:07:53.366 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-26T01:07:53.367 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_max_deferred_txc": "32", 2024-04-26T01:07:53.367 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_min_alloc_size": "0", 2024-04-26T01:07:53.367 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-26T01:07:53.367 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-26T01:07:53.367 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_nid_prealloc": "1024", 2024-04-26T01:07:53.367 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-26T01:07:53.367 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-26T01:07:53.367 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-26T01:07:53.367 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-26T01:07:53.367 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_retry_disk_reads": "3", 2024-04-26T01:07:53.367 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_rocksdb_cf": "true", 2024-04-26T01:07:53.367 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-26T01:07:53.368 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-26T01:07:53.368 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-26T01:07:53.368 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-26T01:07:53.368 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-26T01:07:53.368 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-26T01:07:53.368 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_spdk_mem": "512", 2024-04-26T01:07:53.368 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-26T01:07:53.368 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-26T01:07:53.368 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-26T01:07:53.368 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-26T01:07:53.368 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-26T01:07:53.369 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-26T01:07:53.369 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-26T01:07:53.369 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_tracing": "false", 2024-04-26T01:07:53.369 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-26T01:07:53.369 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-26T01:07:53.369 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-26T01:07:53.369 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-26T01:07:53.369 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-26T01:07:53.369 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-26T01:07:53.369 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-26T01:07:53.369 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-26T01:07:53.369 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-26T01:07:53.369 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_zero_block_detection": "false", 2024-04-26T01:07:53.370 INFO:teuthology.orchestra.run.smithi008.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-26T01:07:53.370 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-26T01:07:53.370 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-26T01:07:53.370 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-26T01:07:53.370 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-26T01:07:53.370 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-26T01:07:53.370 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-26T01:07:53.370 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-26T01:07:53.370 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-26T01:07:53.370 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-26T01:07:53.370 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-26T01:07:53.370 INFO:teuthology.orchestra.run.smithi008.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-26T01:07:53.370 INFO:teuthology.orchestra.run.smithi008.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-26T01:07:53.371 INFO:teuthology.orchestra.run.smithi008.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-26T01:07:53.371 INFO:teuthology.orchestra.run.smithi008.stdout: "cephx_cluster_require_signatures": "false", 2024-04-26T01:07:53.371 INFO:teuthology.orchestra.run.smithi008.stdout: "cephx_cluster_require_version": "2", 2024-04-26T01:07:53.371 INFO:teuthology.orchestra.run.smithi008.stdout: "cephx_require_signatures": "false", 2024-04-26T01:07:53.371 INFO:teuthology.orchestra.run.smithi008.stdout: "cephx_require_version": "2", 2024-04-26T01:07:53.371 INFO:teuthology.orchestra.run.smithi008.stdout: "cephx_service_require_signatures": "false", 2024-04-26T01:07:53.371 INFO:teuthology.orchestra.run.smithi008.stdout: "cephx_service_require_version": "2", 2024-04-26T01:07:53.371 INFO:teuthology.orchestra.run.smithi008.stdout: "cephx_sign_messages": "true", 2024-04-26T01:07:53.371 INFO:teuthology.orchestra.run.smithi008.stdout: "chdir": "", 2024-04-26T01:07:53.371 INFO:teuthology.orchestra.run.smithi008.stdout: "client_acl_type": "", 2024-04-26T01:07:53.371 INFO:teuthology.orchestra.run.smithi008.stdout: "client_asio_thread_count": "2", 2024-04-26T01:07:53.371 INFO:teuthology.orchestra.run.smithi008.stdout: "client_cache_mid": "0.750000", 2024-04-26T01:07:53.372 INFO:teuthology.orchestra.run.smithi008.stdout: "client_cache_size": "16384", 2024-04-26T01:07:53.372 INFO:teuthology.orchestra.run.smithi008.stdout: "client_caps_release_delay": "5", 2024-04-26T01:07:53.372 INFO:teuthology.orchestra.run.smithi008.stdout: "client_check_pool_perm": "true", 2024-04-26T01:07:53.372 INFO:teuthology.orchestra.run.smithi008.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-26T01:07:53.372 INFO:teuthology.orchestra.run.smithi008.stdout: "client_debug_force_sync_read": "false", 2024-04-26T01:07:53.372 INFO:teuthology.orchestra.run.smithi008.stdout: "client_debug_getattr_caps": "false", 2024-04-26T01:07:53.372 INFO:teuthology.orchestra.run.smithi008.stdout: "client_debug_inject_tick_delay": "0", 2024-04-26T01:07:53.372 INFO:teuthology.orchestra.run.smithi008.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-26T01:07:53.372 INFO:teuthology.orchestra.run.smithi008.stdout: "client_die_on_failed_remount": "false", 2024-04-26T01:07:53.372 INFO:teuthology.orchestra.run.smithi008.stdout: "client_dirsize_rbytes": "true", 2024-04-26T01:07:53.373 INFO:teuthology.orchestra.run.smithi008.stdout: "client_force_lazyio": "false", 2024-04-26T01:07:53.373 INFO:teuthology.orchestra.run.smithi008.stdout: "client_fs": "", 2024-04-26T01:07:53.373 INFO:teuthology.orchestra.run.smithi008.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-26T01:07:53.373 INFO:teuthology.orchestra.run.smithi008.stdout: "client_inject_release_failure": "false", 2024-04-26T01:07:53.373 INFO:teuthology.orchestra.run.smithi008.stdout: "client_max_inline_size": "4096", 2024-04-26T01:07:53.373 INFO:teuthology.orchestra.run.smithi008.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-26T01:07:53.373 INFO:teuthology.orchestra.run.smithi008.stdout: "client_mds_namespace": "", 2024-04-26T01:07:53.373 INFO:teuthology.orchestra.run.smithi008.stdout: "client_metadata": "", 2024-04-26T01:07:53.373 INFO:teuthology.orchestra.run.smithi008.stdout: "client_mount_gid": "-1", 2024-04-26T01:07:53.373 INFO:teuthology.orchestra.run.smithi008.stdout: "client_mount_timeout": "300", 2024-04-26T01:07:53.373 INFO:teuthology.orchestra.run.smithi008.stdout: "client_mount_uid": "-1", 2024-04-26T01:07:53.373 INFO:teuthology.orchestra.run.smithi008.stdout: "client_mountpoint": "/", 2024-04-26T01:07:53.373 INFO:teuthology.orchestra.run.smithi008.stdout: "client_notify_timeout": "10", 2024-04-26T01:07:53.374 INFO:teuthology.orchestra.run.smithi008.stdout: "client_oc": "true", 2024-04-26T01:07:53.374 INFO:teuthology.orchestra.run.smithi008.stdout: "client_oc_max_dirty": "104857600", 2024-04-26T01:07:53.374 INFO:teuthology.orchestra.run.smithi008.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-26T01:07:53.374 INFO:teuthology.orchestra.run.smithi008.stdout: "client_oc_max_objects": "1000", 2024-04-26T01:07:53.374 INFO:teuthology.orchestra.run.smithi008.stdout: "client_oc_size": "209715200", 2024-04-26T01:07:53.374 INFO:teuthology.orchestra.run.smithi008.stdout: "client_oc_target_dirty": "8388608", 2024-04-26T01:07:53.374 INFO:teuthology.orchestra.run.smithi008.stdout: "client_permissions": "true", 2024-04-26T01:07:53.374 INFO:teuthology.orchestra.run.smithi008.stdout: "client_quota": "true", 2024-04-26T01:07:53.374 INFO:teuthology.orchestra.run.smithi008.stdout: "client_quota_df": "true", 2024-04-26T01:07:53.374 INFO:teuthology.orchestra.run.smithi008.stdout: "client_readahead_max_bytes": "0", 2024-04-26T01:07:53.374 INFO:teuthology.orchestra.run.smithi008.stdout: "client_readahead_max_periods": "4", 2024-04-26T01:07:53.374 INFO:teuthology.orchestra.run.smithi008.stdout: "client_readahead_min": "131072", 2024-04-26T01:07:53.374 INFO:teuthology.orchestra.run.smithi008.stdout: "client_reconnect_stale": "false", 2024-04-26T01:07:53.375 INFO:teuthology.orchestra.run.smithi008.stdout: "client_shutdown_timeout": "30", 2024-04-26T01:07:53.375 INFO:teuthology.orchestra.run.smithi008.stdout: "client_snapdir": ".snap", 2024-04-26T01:07:53.375 INFO:teuthology.orchestra.run.smithi008.stdout: "client_tick_interval": "1", 2024-04-26T01:07:53.375 INFO:teuthology.orchestra.run.smithi008.stdout: "client_trace": "", 2024-04-26T01:07:53.375 INFO:teuthology.orchestra.run.smithi008.stdout: "client_try_dentry_invalidate": "false", 2024-04-26T01:07:53.375 INFO:teuthology.orchestra.run.smithi008.stdout: "client_use_faked_inos": "false", 2024-04-26T01:07:53.375 INFO:teuthology.orchestra.run.smithi008.stdout: "client_use_random_mds": "false", 2024-04-26T01:07:53.375 INFO:teuthology.orchestra.run.smithi008.stdout: "clog_to_graylog": "false", 2024-04-26T01:07:53.375 INFO:teuthology.orchestra.run.smithi008.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-26T01:07:53.375 INFO:teuthology.orchestra.run.smithi008.stdout: "clog_to_graylog_port": "12201", 2024-04-26T01:07:53.375 INFO:teuthology.orchestra.run.smithi008.stdout: "clog_to_monitors": "default=true", 2024-04-26T01:07:53.375 INFO:teuthology.orchestra.run.smithi008.stdout: "clog_to_syslog": "false", 2024-04-26T01:07:53.375 INFO:teuthology.orchestra.run.smithi008.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-26T01:07:53.375 INFO:teuthology.orchestra.run.smithi008.stdout: "clog_to_syslog_level": "info", 2024-04-26T01:07:53.376 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "-", 2024-04-26T01:07:53.376 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_network": "", 2024-04-26T01:07:53.376 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_network_interface": "", 2024-04-26T01:07:53.376 INFO:teuthology.orchestra.run.smithi008.stdout: "compressor_zlib_isal": "false", 2024-04-26T01:07:53.376 INFO:teuthology.orchestra.run.smithi008.stdout: "compressor_zlib_level": "5", 2024-04-26T01:07:53.376 INFO:teuthology.orchestra.run.smithi008.stdout: "compressor_zlib_winsize": "-15", 2024-04-26T01:07:53.376 INFO:teuthology.orchestra.run.smithi008.stdout: "compressor_zstd_level": "1", 2024-04-26T01:07:53.376 INFO:teuthology.orchestra.run.smithi008.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-26T01:07:53.376 INFO:teuthology.orchestra.run.smithi008.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-26T01:07:53.376 INFO:teuthology.orchestra.run.smithi008.stdout: "crimson_alien_op_num_threads": "6", 2024-04-26T01:07:53.376 INFO:teuthology.orchestra.run.smithi008.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-26T01:07:53.376 INFO:teuthology.orchestra.run.smithi008.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-26T01:07:53.376 INFO:teuthology.orchestra.run.smithi008.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "crimson_seastar_cpu_cores": "", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_location": "", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_location_hook": "", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_location_hook_timeout": "10", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "daemonize": "false", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "daos_pool": "tank", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_alienstore": "0/5", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_allow_any_pool_priority": "false", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_asok": "1/5", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_asok_assert_abort": "false", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_asserts_on_shutdown": "true", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_auth": "20/20", 2024-04-26T01:07:53.377 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_bdev": "1/3", 2024-04-26T01:07:53.378 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_bluefs": "1/5", 2024-04-26T01:07:53.378 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_bluestore": "1/5", 2024-04-26T01:07:53.378 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_buffer": "0/1", 2024-04-26T01:07:53.378 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_ceph_exporter": "1/5", 2024-04-26T01:07:53.378 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_cephfs_mirror": "0/5", 2024-04-26T01:07:53.378 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_cephsqlite": "0/5", 2024-04-26T01:07:53.378 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_client": "0/5", 2024-04-26T01:07:53.378 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_compressor": "1/5", 2024-04-26T01:07:53.378 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_context": "0/1", 2024-04-26T01:07:53.378 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_crimson_interrupt": "0/5", 2024-04-26T01:07:53.378 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_crush": "1/1", 2024-04-26T01:07:53.378 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_crypto": "1/5", 2024-04-26T01:07:53.378 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_cyanstore": "0/5", 2024-04-26T01:07:53.378 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_deliberately_leak_memory": "false", 2024-04-26T01:07:53.378 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_disable_randomized_ping": "false", 2024-04-26T01:07:53.379 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_dpdk": "1/5", 2024-04-26T01:07:53.379 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_eventtrace": "1/5", 2024-04-26T01:07:53.379 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_filer": "0/1", 2024-04-26T01:07:53.379 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_filestore": "1/3", 2024-04-26T01:07:53.379 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_finisher": "1/1", 2024-04-26T01:07:53.379 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_fuse": "1/5", 2024-04-26T01:07:53.379 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_heartbeat_testing_span": "0", 2024-04-26T01:07:53.379 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_heartbeatmap": "1/5", 2024-04-26T01:07:53.379 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-26T01:07:53.379 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_javaclient": "1/5", 2024-04-26T01:07:53.379 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_journal": "1/3", 2024-04-26T01:07:53.379 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_journaler": "0/5", 2024-04-26T01:07:53.379 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_kstore": "1/5", 2024-04-26T01:07:53.380 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_lockdep": "0/1", 2024-04-26T01:07:53.380 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mclock": "1/5", 2024-04-26T01:07:53.380 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mds": "1/5", 2024-04-26T01:07:53.380 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mds_balancer": "1/5", 2024-04-26T01:07:53.380 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mds_locker": "1/5", 2024-04-26T01:07:53.380 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mds_log": "1/5", 2024-04-26T01:07:53.380 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mds_log_expire": "1/5", 2024-04-26T01:07:53.380 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mds_migrator": "1/5", 2024-04-26T01:07:53.380 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mds_quiesce": "3/5", 2024-04-26T01:07:53.380 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_memstore": "1/5", 2024-04-26T01:07:53.380 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mgr": "2/5", 2024-04-26T01:07:53.380 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mgrc": "1/5", 2024-04-26T01:07:53.381 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mon": "20/20", 2024-04-26T01:07:53.381 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_monc": "0/10", 2024-04-26T01:07:53.381 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_ms": "1/1", 2024-04-26T01:07:53.381 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_none": "0/5", 2024-04-26T01:07:53.381 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_objclass": "0/5", 2024-04-26T01:07:53.381 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_objectcacher": "0/5", 2024-04-26T01:07:53.381 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_objecter": "0/1", 2024-04-26T01:07:53.381 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_optracker": "0/5", 2024-04-26T01:07:53.381 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_osd": "1/5", 2024-04-26T01:07:53.381 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_paxos": "20/20", 2024-04-26T01:07:53.381 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_perfcounter": "1/5", 2024-04-26T01:07:53.381 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_prioritycache": "1/5", 2024-04-26T01:07:53.382 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rados": "0/5", 2024-04-26T01:07:53.382 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rbd": "0/5", 2024-04-26T01:07:53.382 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rbd_mirror": "0/5", 2024-04-26T01:07:53.382 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rbd_pwl": "0/5", 2024-04-26T01:07:53.382 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rbd_replay": "0/5", 2024-04-26T01:07:53.382 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_refs": "0/0", 2024-04-26T01:07:53.382 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_reserver": "1/1", 2024-04-26T01:07:53.382 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rgw": "1/5", 2024-04-26T01:07:53.382 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rgw_access": "1/5", 2024-04-26T01:07:53.382 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rgw_datacache": "1/5", 2024-04-26T01:07:53.382 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rgw_dbstore": "1/5", 2024-04-26T01:07:53.382 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rgw_flight": "1/5", 2024-04-26T01:07:53.382 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-26T01:07:53.382 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rgw_sync": "1/5", 2024-04-26T01:07:53.383 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rocksdb": "4/5", 2024-04-26T01:07:53.383 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore": "0/5", 2024-04-26T01:07:53.383 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_backref": "0/5", 2024-04-26T01:07:53.383 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_cache": "0/5", 2024-04-26T01:07:53.383 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_cleaner": "0/5", 2024-04-26T01:07:53.383 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_device": "0/5", 2024-04-26T01:07:53.383 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_epm": "0/5", 2024-04-26T01:07:53.383 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-26T01:07:53.383 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_journal": "0/5", 2024-04-26T01:07:53.383 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_lba": "0/5", 2024-04-26T01:07:53.383 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_odata": "0/5", 2024-04-26T01:07:53.383 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_omap": "0/5", 2024-04-26T01:07:53.383 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_onode": "0/5", 2024-04-26T01:07:53.383 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_t": "0/5", 2024-04-26T01:07:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_tm": "0/5", 2024-04-26T01:07:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_striper": "0/1", 2024-04-26T01:07:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_test": "0/5", 2024-04-26T01:07:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_throttle": "1/1", 2024-04-26T01:07:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_timer": "0/1", 2024-04-26T01:07:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_tp": "0/5", 2024-04-26T01:07:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_trace": "1/5", 2024-04-26T01:07:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-26T01:07:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: "device_failure_prediction_mode": "none", 2024-04-26T01:07:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-26T01:07:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-04-26T01:07:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: "err_to_graylog": "false", 2024-04-26T01:07:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: "err_to_journald": "false", 2024-04-26T01:07:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: "err_to_stderr": "true", 2024-04-26T01:07:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: "err_to_syslog": "false", 2024-04-26T01:07:53.385 INFO:teuthology.orchestra.run.smithi008.stdout: "event_tracing": "false", 2024-04-26T01:07:53.385 INFO:teuthology.orchestra.run.smithi008.stdout: "exporter_addr": "0.0.0.0", 2024-04-26T01:07:53.385 INFO:teuthology.orchestra.run.smithi008.stdout: "exporter_http_port": "9926", 2024-04-26T01:07:53.385 INFO:teuthology.orchestra.run.smithi008.stdout: "exporter_prio_limit": "5", 2024-04-26T01:07:53.385 INFO:teuthology.orchestra.run.smithi008.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-26T01:07:53.385 INFO:teuthology.orchestra.run.smithi008.stdout: "exporter_sort_metrics": "true", 2024-04-26T01:07:53.385 INFO:teuthology.orchestra.run.smithi008.stdout: "exporter_stats_period": "5", 2024-04-26T01:07:53.385 INFO:teuthology.orchestra.run.smithi008.stdout: "fake_statfs_for_testing": "0", 2024-04-26T01:07:53.385 INFO:teuthology.orchestra.run.smithi008.stdout: "fatal_signal_handlers": "true", 2024-04-26T01:07:53.385 INFO:teuthology.orchestra.run.smithi008.stdout: "filer_max_purge_ops": "10", 2024-04-26T01:07:53.385 INFO:teuthology.orchestra.run.smithi008.stdout: "filer_max_truncate_ops": "128", 2024-04-26T01:07:53.385 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_apply_finisher_threads": "1", 2024-04-26T01:07:53.386 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_blackhole": "false", 2024-04-26T01:07:53.386 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_btrfs_clone_range": "true", 2024-04-26T01:07:53.386 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_btrfs_snap": "true", 2024-04-26T01:07:53.386 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_caller_concurrency": "10", 2024-04-26T01:07:53.386 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_collect_device_partition_information": "true", 2024-04-26T01:07:53.386 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_commit_timeout": "600.000000", 2024-04-26T01:07:53.386 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_debug_inject_read_err": "false", 2024-04-26T01:07:53.386 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_debug_omap_check": "false", 2024-04-26T01:07:53.386 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_debug_random_read_err": "0.000000", 2024-04-26T01:07:53.386 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_debug_verify_split": "false", 2024-04-26T01:07:53.386 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_dump_file": "", 2024-04-26T01:07:53.386 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-26T01:07:53.386 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-26T01:07:53.387 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_fadvise": "true", 2024-04-26T01:07:53.387 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_fail_eio": "true", 2024-04-26T01:07:53.387 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_fd_cache_shards": "16", 2024-04-26T01:07:53.387 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_fd_cache_size": "128", 2024-04-26T01:07:53.387 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_fiemap": "false", 2024-04-26T01:07:53.387 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_fiemap_threshold": "4096", 2024-04-26T01:07:53.387 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-26T01:07:53.387 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-26T01:07:53.387 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_inject_stall": "0", 2024-04-26T01:07:53.387 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_journal_parallel": "false", 2024-04-26T01:07:53.387 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_journal_trailing": "false", 2024-04-26T01:07:53.387 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_journal_writeahead": "false", 2024-04-26T01:07:53.387 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_kill_at": "0", 2024-04-26T01:07:53.387 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-26T01:07:53.388 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-26T01:07:53.388 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-26T01:07:53.388 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-26T01:07:53.388 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-26T01:07:53.388 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattrs": "0", 2024-04-26T01:07:53.388 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-26T01:07:53.388 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-26T01:07:53.388 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-26T01:07:53.388 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-26T01:07:53.388 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_xattr_value_size": "0", 2024-04-26T01:07:53.388 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-26T01:07:53.388 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-26T01:07:53.388 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-26T01:07:53.388 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_merge_threshold": "-10", 2024-04-26T01:07:53.389 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-26T01:07:53.389 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_odsync_write": "false", 2024-04-26T01:07:53.389 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_omap_backend": "rocksdb", 2024-04-26T01:07:53.389 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_omap_backend_path": "", 2024-04-26T01:07:53.389 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-26T01:07:53.389 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-26T01:07:53.389 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-26T01:07:53.389 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_op_thread_timeout": "60", 2024-04-26T01:07:53.389 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_op_threads": "2", 2024-04-26T01:07:53.389 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_punch_hole": "false", 2024-04-26T01:07:53.389 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-26T01:07:53.389 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-26T01:07:53.389 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-26T01:07:53.390 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-26T01:07:53.390 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-26T01:07:53.390 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-26T01:07:53.390 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-26T01:07:53.390 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-26T01:07:53.390 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-26T01:07:53.390 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_max_ops": "50", 2024-04-26T01:07:53.390 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-26T01:07:53.390 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_seek_data_hole": "false", 2024-04-26T01:07:53.390 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_sloppy_crc": "false", 2024-04-26T01:07:53.390 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-26T01:07:53.390 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_splice": "false", 2024-04-26T01:07:53.390 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_split_multiple": "2", 2024-04-26T01:07:53.391 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_split_rand_factor": "20", 2024-04-26T01:07:53.391 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_update_to": "1000", 2024-04-26T01:07:53.391 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-26T01:07:53.391 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-26T01:07:53.391 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-26T01:07:53.391 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-26T01:07:53.391 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-26T01:07:53.391 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-26T01:07:53.391 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_enable": "true", 2024-04-26T01:07:53.391 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-26T01:07:53.391 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-26T01:07:53.391 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-26T01:07:53.391 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-26T01:07:53.392 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-26T01:07:53.392 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-26T01:07:53.392 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_xfs_extsize": "false", 2024-04-26T01:07:53.392 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_zfs_snap": "false", 2024-04-26T01:07:53.392 INFO:teuthology.orchestra.run.smithi008.stdout: "fio_dir": "/tmp/fio", 2024-04-26T01:07:53.392 INFO:teuthology.orchestra.run.smithi008.stdout: "fsid": "4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554", 2024-04-26T01:07:53.392 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_allow_other": "true", 2024-04-26T01:07:53.392 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_atomic_o_trunc": "true", 2024-04-26T01:07:53.392 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_big_writes": "true", 2024-04-26T01:07:53.392 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_debug": "false", 2024-04-26T01:07:53.392 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_default_permissions": "false", 2024-04-26T01:07:53.392 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_disable_pagecache": "false", 2024-04-26T01:07:53.392 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_max_write": "0", 2024-04-26T01:07:53.392 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_multithreaded": "true", 2024-04-26T01:07:53.393 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_require_active_mds": "true", 2024-04-26T01:07:53.393 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_set_user_groups": "true", 2024-04-26T01:07:53.393 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_splice_move": "true", 2024-04-26T01:07:53.393 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_splice_read": "true", 2024-04-26T01:07:53.393 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_splice_write": "true", 2024-04-26T01:07:53.393 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-26T01:07:53.393 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_use_invalidate_cb": "true", 2024-04-26T01:07:53.393 INFO:teuthology.orchestra.run.smithi008.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-26T01:07:53.393 INFO:teuthology.orchestra.run.smithi008.stdout: "gss_target_name": "ceph", 2024-04-26T01:07:53.393 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_file": "", 2024-04-26T01:07:53.393 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_inject_failure": "0", 2024-04-26T01:07:53.393 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_interval": "5", 2024-04-26T01:07:53.393 INFO:teuthology.orchestra.run.smithi008.stdout: "host": "smithi008", 2024-04-26T01:07:53.394 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-26T01:07:53.394 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-26T01:07:53.394 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-26T01:07:53.394 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-26T01:07:53.394 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-26T01:07:53.394 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-26T01:07:53.394 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-26T01:07:53.394 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-26T01:07:53.394 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-26T01:07:53.394 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-26T01:07:53.394 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-26T01:07:53.394 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-26T01:07:53.394 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-26T01:07:53.394 INFO:teuthology.orchestra.run.smithi008.stdout: "inject_early_sigterm": "false", 2024-04-26T01:07:53.395 INFO:teuthology.orchestra.run.smithi008.stdout: "jaeger_agent_port": "6799", 2024-04-26T01:07:53.395 INFO:teuthology.orchestra.run.smithi008.stdout: "jaeger_tracing_enable": "false", 2024-04-26T01:07:53.395 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_aio": "true", 2024-04-26T01:07:53.395 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_align_min_size": "65536", 2024-04-26T01:07:53.395 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_block_align": "true", 2024-04-26T01:07:53.395 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_block_size": "4096", 2024-04-26T01:07:53.395 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_dio": "true", 2024-04-26T01:07:53.395 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_discard": "false", 2024-04-26T01:07:53.395 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_force_aio": "false", 2024-04-26T01:07:53.395 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_ignore_corruption": "false", 2024-04-26T01:07:53.395 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_max_write_bytes": "10485760", 2024-04-26T01:07:53.395 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_max_write_entries": "100", 2024-04-26T01:07:53.395 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_replay_from": "0", 2024-04-26T01:07:53.395 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-26T01:07:53.395 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-26T01:07:53.396 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-26T01:07:53.396 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-26T01:07:53.396 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_write_header_frequency": "0", 2024-04-26T01:07:53.396 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_zero_on_create": "false", 2024-04-26T01:07:53.396 INFO:teuthology.orchestra.run.smithi008.stdout: "journaler_prefetch_periods": "10", 2024-04-26T01:07:53.396 INFO:teuthology.orchestra.run.smithi008.stdout: "journaler_prezero_periods": "5", 2024-04-26T01:07:53.396 INFO:teuthology.orchestra.run.smithi008.stdout: "journaler_write_head_interval": "15", 2024-04-26T01:07:53.396 INFO:teuthology.orchestra.run.smithi008.stdout: "key": "", 2024-04-26T01:07:53.396 INFO:teuthology.orchestra.run.smithi008.stdout: "keyfile": "", 2024-04-26T01:07:53.396 INFO:teuthology.orchestra.run.smithi008.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-26T01:07:53.396 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_backend": "rocksdb", 2024-04-26T01:07:53.396 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_default_stripe_size": "65536", 2024-04-26T01:07:53.396 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_fsck_on_mount": "false", 2024-04-26T01:07:53.396 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-26T01:07:53.396 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_max_bytes": "67108864", 2024-04-26T01:07:53.397 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_max_ops": "512", 2024-04-26T01:07:53.397 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_nid_prealloc": "1024", 2024-04-26T01:07:53.397 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_onode_map_size": "1024", 2024-04-26T01:07:53.397 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-26T01:07:53.397 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_sync_submit_transaction": "false", 2024-04-26T01:07:53.397 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_sync_transaction": "false", 2024-04-26T01:07:53.397 INFO:teuthology.orchestra.run.smithi008.stdout: "librados_thread_count": "2", 2024-04-26T01:07:53.397 INFO:teuthology.orchestra.run.smithi008.stdout: "lockdep": "false", 2024-04-26T01:07:53.397 INFO:teuthology.orchestra.run.smithi008.stdout: "lockdep_force_backtrace": "false", 2024-04-26T01:07:53.397 INFO:teuthology.orchestra.run.smithi008.stdout: "log_coarse_timestamps": "true", 2024-04-26T01:07:53.397 INFO:teuthology.orchestra.run.smithi008.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-26T01:07:53.397 INFO:teuthology.orchestra.run.smithi008.stdout: "log_flush_on_exit": "false", 2024-04-26T01:07:53.397 INFO:teuthology.orchestra.run.smithi008.stdout: "log_graylog_host": "127.0.0.1", 2024-04-26T01:07:53.397 INFO:teuthology.orchestra.run.smithi008.stdout: "log_graylog_port": "12201", 2024-04-26T01:07:53.398 INFO:teuthology.orchestra.run.smithi008.stdout: "log_max_new": "1000", 2024-04-26T01:07:53.398 INFO:teuthology.orchestra.run.smithi008.stdout: "log_max_recent": "10000", 2024-04-26T01:07:53.398 INFO:teuthology.orchestra.run.smithi008.stdout: "log_stderr_prefix": "", 2024-04-26T01:07:53.398 INFO:teuthology.orchestra.run.smithi008.stdout: "log_stop_at_utilization": "0.970000", 2024-04-26T01:07:53.398 INFO:teuthology.orchestra.run.smithi008.stdout: "log_to_file": "true", 2024-04-26T01:07:53.398 INFO:teuthology.orchestra.run.smithi008.stdout: "log_to_graylog": "false", 2024-04-26T01:07:53.398 INFO:teuthology.orchestra.run.smithi008.stdout: "log_to_journald": "false", 2024-04-26T01:07:53.398 INFO:teuthology.orchestra.run.smithi008.stdout: "log_to_stderr": "false", 2024-04-26T01:07:53.398 INFO:teuthology.orchestra.run.smithi008.stdout: "log_to_syslog": "false", 2024-04-26T01:07:53.398 INFO:teuthology.orchestra.run.smithi008.stdout: "max_rotating_auth_attempts": "10", 2024-04-26T01:07:53.398 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-26T01:07:53.398 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_action_on_write_error": "1", 2024-04-26T01:07:53.398 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_alternate_name_max": "8192", 2024-04-26T01:07:53.398 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_asio_thread_count": "2", 2024-04-26T01:07:53.399 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_export_pin": "true", 2024-04-26T01:07:53.399 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_fragment_dirs": "true", 2024-04-26T01:07:53.399 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-26T01:07:53.399 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_fragment_interval": "5", 2024-04-26T01:07:53.399 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-26T01:07:53.399 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-26T01:07:53.399 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_interval": "10", 2024-04-26T01:07:53.399 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_max": "-1", 2024-04-26T01:07:53.399 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_max_until": "-1", 2024-04-26T01:07:53.399 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_merge_size": "50", 2024-04-26T01:07:53.399 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_midchunk": "0.300000", 2024-04-26T01:07:53.399 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-26T01:07:53.399 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_min_start": "0.200000", 2024-04-26T01:07:53.399 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_minchunk": "0.001000", 2024-04-26T01:07:53.400 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_mode": "0", 2024-04-26T01:07:53.400 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_need_max": "1.200000", 2024-04-26T01:07:53.400 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_need_min": "0.800000", 2024-04-26T01:07:53.400 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_overload_epochs": "2", 2024-04-26T01:07:53.400 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-26T01:07:53.400 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-26T01:07:53.400 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_split_bits": "3", 2024-04-26T01:07:53.400 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-26T01:07:53.400 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_split_size": "10000", 2024-04-26T01:07:53.400 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-26T01:07:53.400 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_target_decay": "10.000000", 2024-04-26T01:07:53.400 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-26T01:07:53.400 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_beacon_grace": "15.000000", 2024-04-26T01:07:53.400 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_beacon_interval": "4.000000", 2024-04-26T01:07:53.400 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-26T01:07:53.401 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-26T01:07:53.401 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_mid": "0.700000", 2024-04-26T01:07:53.401 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-26T01:07:53.401 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_quiesce_delay": "0", 2024-04-26T01:07:53.401 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-26T01:07:53.401 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-26T01:07:53.401 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-26T01:07:53.401 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_release_free_interval": "10", 2024-04-26T01:07:53.401 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_reservation": "0.050000", 2024-04-26T01:07:53.401 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-26T01:07:53.401 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_trim_interval": "1", 2024-04-26T01:07:53.401 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_trim_threshold": "262144", 2024-04-26T01:07:53.401 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-26T01:07:53.401 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-26T01:07:53.401 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-26T01:07:53.402 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_client_prealloc_inos": "1000", 2024-04-26T01:07:53.402 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-26T01:07:53.402 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_connect_bootstrapping": "false", 2024-04-26T01:07:53.402 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_damage_table_max_entries": "10000", 2024-04-26T01:07:53.402 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-26T01:07:53.402 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_debug_auth_pins": "false", 2024-04-26T01:07:53.402 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_debug_frag": "false", 2024-04-26T01:07:53.402 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_debug_scatterstat": "false", 2024-04-26T01:07:53.402 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_debug_subtrees": "false", 2024-04-26T01:07:53.402 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_decay_halflife": "5.000000", 2024-04-26T01:07:53.402 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_default_dir_hash": "2", 2024-04-26T01:07:53.402 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_defer_session_stale": "true", 2024-04-26T01:07:53.402 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_deny_all_reconnect": "false", 2024-04-26T01:07:53.403 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dir_keys_per_op": "16384", 2024-04-26T01:07:53.403 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dir_max_commit_size": "10", 2024-04-26T01:07:53.403 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dir_max_entries": "0", 2024-04-26T01:07:53.403 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dir_prefetch": "true", 2024-04-26T01:07:53.403 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-26T01:07:53.403 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-26T01:07:53.403 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dump_cache_on_map": "false", 2024-04-26T01:07:53.403 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-26T01:07:53.403 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-26T01:07:53.403 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_early_reply": "true", 2024-04-26T01:07:53.403 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_enable_op_tracker": "true", 2024-04-26T01:07:53.403 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_enforce_unique_name": "true", 2024-04-26T01:07:53.403 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-26T01:07:53.403 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-26T01:07:53.404 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_export_ephemeral_random": "true", 2024-04-26T01:07:53.404 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-26T01:07:53.404 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-26T01:07:53.404 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-26T01:07:53.404 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-26T01:07:53.404 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-26T01:07:53.404 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-26T01:07:53.404 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-26T01:07:53.404 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-26T01:07:53.404 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_health_summarize_threshold": "10", 2024-04-26T01:07:53.404 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-26T01:07:53.404 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-26T01:07:53.404 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_inject_health_dummy": "false", 2024-04-26T01:07:53.405 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-26T01:07:53.405 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_inject_migrator_session_race": "false", 2024-04-26T01:07:53.405 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-26T01:07:53.405 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-26T01:07:53.405 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-26T01:07:53.405 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_join_fs": "", 2024-04-26T01:07:53.405 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_journal_format": "1", 2024-04-26T01:07:53.405 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_create_at": "0", 2024-04-26T01:07:53.405 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_export_at": "0", 2024-04-26T01:07:53.405 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_import_at": "0", 2024-04-26T01:07:53.405 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_journal_at": "0", 2024-04-26T01:07:53.405 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_journal_expire_at": "0", 2024-04-26T01:07:53.405 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_journal_replay_at": "0", 2024-04-26T01:07:53.405 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_link_at": "0", 2024-04-26T01:07:53.405 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_mdstable_at": "0", 2024-04-26T01:07:53.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_openc_at": "0", 2024-04-26T01:07:53.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_rename_at": "0", 2024-04-26T01:07:53.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_shutdown_at": "0", 2024-04-26T01:07:53.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-26T01:07:53.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_event_large_threshold": "524288", 2024-04-26T01:07:53.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_events_per_segment": "1024", 2024-04-26T01:07:53.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-26T01:07:53.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_max_events": "-1", 2024-04-26T01:07:53.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_max_segments": "128", 2024-04-26T01:07:53.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_pause": "false", 2024-04-26T01:07:53.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_segment_size": "0", 2024-04-26T01:07:53.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-26T01:07:53.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-26T01:07:53.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-26T01:07:53.407 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_trim_threshold": "128", 2024-04-26T01:07:53.407 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-26T01:07:53.407 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_warn_factor": "2.000000", 2024-04-26T01:07:53.407 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_caps_per_client": "1048576", 2024-04-26T01:07:53.407 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_completed_flushes": "100000", 2024-04-26T01:07:53.407 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_completed_requests": "100000", 2024-04-26T01:07:53.407 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_export_size": "20971520", 2024-04-26T01:07:53.407 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_file_recover": "32", 2024-04-26T01:07:53.407 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_purge_files": "64", 2024-04-26T01:07:53.407 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_purge_ops": "8192", 2024-04-26T01:07:53.407 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-26T01:07:53.407 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-26T01:07:53.407 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_snaps_per_dir": "100", 2024-04-26T01:07:53.407 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_metrics_update_interval": "2", 2024-04-26T01:07:53.407 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_min_caps_per_client": "100", 2024-04-26T01:07:53.408 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_min_caps_working_set": "10000", 2024-04-26T01:07:53.408 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-26T01:07:53.408 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_numa_node": "-1", 2024-04-26T01:07:53.408 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-26T01:07:53.408 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_op_complaint_time": "30.000000", 2024-04-26T01:07:53.408 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_op_history_duration": "600", 2024-04-26T01:07:53.408 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_op_history_size": "20", 2024-04-26T01:07:53.408 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_op_history_slow_op_size": "20", 2024-04-26T01:07:53.408 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-26T01:07:53.408 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_op_log_threshold": "5", 2024-04-26T01:07:53.408 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_ping_grace": "15", 2024-04-26T01:07:53.408 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_ping_interval": "5", 2024-04-26T01:07:53.408 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-26T01:07:53.408 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-26T01:07:53.409 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_recall_max_caps": "30000", 2024-04-26T01:07:53.409 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-26T01:07:53.409 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-26T01:07:53.409 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-26T01:07:53.409 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_recall_warning_threshold": "262144", 2024-04-26T01:07:53.409 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-26T01:07:53.409 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_replay_interval": "1.000000", 2024-04-26T01:07:53.409 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-26T01:07:53.409 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-26T01:07:53.409 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_root_ino_gid": "0", 2024-04-26T01:07:53.409 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_root_ino_uid": "0", 2024-04-26T01:07:53.409 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-26T01:07:53.409 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-26T01:07:53.409 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-26T01:07:53.410 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-26T01:07:53.410 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-26T01:07:53.410 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-26T01:07:53.410 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-26T01:07:53.410 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-26T01:07:53.410 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-26T01:07:53.410 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-26T01:07:53.410 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_shutdown_check": "0", 2024-04-26T01:07:53.410 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_skip_ino": "0", 2024-04-26T01:07:53.410 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-26T01:07:53.410 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_snap_max_uid": "4294967294", 2024-04-26T01:07:53.410 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_snap_min_uid": "0", 2024-04-26T01:07:53.410 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_snap_rstat": "false", 2024-04-26T01:07:53.410 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_standby_replay_damaged": "false", 2024-04-26T01:07:53.411 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_symlink_recovery": "true", 2024-04-26T01:07:53.411 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-26T01:07:53.411 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_thrash_exports": "0", 2024-04-26T01:07:53.411 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_thrash_fragments": "0", 2024-04-26T01:07:53.411 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_tick_interval": "5.000000", 2024-04-26T01:07:53.411 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_valgrind_exit": "false", 2024-04-26T01:07:53.411 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_verify_backtrace": "1", 2024-04-26T01:07:53.411 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_verify_scatter": "false", 2024-04-26T01:07:53.411 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-26T01:07:53.411 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_wipe_sessions": "false", 2024-04-26T01:07:53.411 INFO:teuthology.orchestra.run.smithi008.stdout: "mempool_debug": "false", 2024-04-26T01:07:53.411 INFO:teuthology.orchestra.run.smithi008.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-26T01:07:53.412 INFO:teuthology.orchestra.run.smithi008.stdout: "memstore_device_bytes": "1073741824", 2024-04-26T01:07:53.412 INFO:teuthology.orchestra.run.smithi008.stdout: "memstore_page_set": "false", 2024-04-26T01:07:53.412 INFO:teuthology.orchestra.run.smithi008.stdout: "memstore_page_size": "65536", 2024-04-26T01:07:53.412 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_client_bytes": "134217728", 2024-04-26T01:07:53.412 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_client_messages": "512", 2024-04-26T01:07:53.412 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-26T01:07:53.412 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-26T01:07:53.412 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-26T01:07:53.412 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-26T01:07:53.412 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_disabled_modules": "", 2024-04-26T01:07:53.412 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-26T01:07:53.412 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_max_pg_num_change": "128", 2024-04-26T01:07:53.412 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_mds_bytes": "134217728", 2024-04-26T01:07:53.412 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_mds_messages": "128", 2024-04-26T01:07:53.412 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-26T01:07:53.413 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_mon_bytes": "134217728", 2024-04-26T01:07:53.413 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_mon_messages": "128", 2024-04-26T01:07:53.413 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_osd_bytes": "536870912", 2024-04-26T01:07:53.413 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_osd_messages": "8192", 2024-04-26T01:07:53.413 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_pool": "true", 2024-04-26T01:07:53.413 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-26T01:07:53.413 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_standby_modules": "true", 2024-04-26T01:07:53.413 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_stats_period": "5", 2024-04-26T01:07:53.413 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_stats_threshold": "5", 2024-04-26T01:07:53.413 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_tick_period": "2", 2024-04-26T01:07:53.413 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-26T01:07:53.413 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-26T01:07:53.413 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_allow_pool_delete": "true", 2024-04-26T01:07:53.413 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_allow_pool_size_one": "true", 2024-04-26T01:07:53.413 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_auth_validate_all_caps": "true", 2024-04-26T01:07:53.414 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-26T01:07:53.414 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-26T01:07:53.414 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_bytes": "104857600", 2024-04-26T01:07:53.414 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_directed_command_retry": "5", 2024-04-26T01:07:53.414 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-26T01:07:53.414 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-26T01:07:53.414 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-26T01:07:53.414 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-26T01:07:53.414 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_hunt_parallel": "3", 2024-04-26T01:07:53.414 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_log_interval": "1.000000", 2024-04-26T01:07:53.414 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-26T01:07:53.414 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_ping_interval": "10.000000", 2024-04-26T01:07:53.414 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-26T01:07:53.414 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_target_rank": "-1", 2024-04-26T01:07:53.415 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-26T01:07:53.415 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-26T01:07:53.415 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-26T01:07:53.415 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_file_level": "debug", 2024-04-26T01:07:53.415 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_file": "true", 2024-04-26T01:07:53.415 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-26T01:07:53.415 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-26T01:07:53.415 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-26T01:07:53.415 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_journald": "false", 2024-04-26T01:07:53.415 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-26T01:07:53.415 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-26T01:07:53.415 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-26T01:07:53.415 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-26T01:07:53.415 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_compact_on_bootstrap": "false", 2024-04-26T01:07:53.416 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_compact_on_start": "false", 2024-04-26T01:07:53.416 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_compact_on_trim": "true", 2024-04-26T01:07:53.416 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-26T01:07:53.416 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-26T01:07:53.416 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-26T01:07:53.416 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cpu_threads": "4", 2024-04-26T01:07:53.416 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_crush_min_required_version": "hammer", 2024-04-26T01:07:53.416 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_daemon_bytes": "419430400", 2024-04-26T01:07:53.416 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-26T01:07:53.416 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_data_avail_crit": "5", 2024-04-26T01:07:53.416 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_data_avail_warn": "5", 2024-04-26T01:07:53.416 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_data_size_warn": "16106127360", 2024-04-26T01:07:53.416 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-26T01:07:53.417 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-26T01:07:53.417 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_dump_json": "false", 2024-04-26T01:07:53.417 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-26T01:07:53.417 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_dump_transactions": "false", 2024-04-26T01:07:53.417 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_extra_checks": "false", 2024-04-26T01:07:53.417 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-26T01:07:53.417 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-26T01:07:53.417 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_no_require_reef": "false", 2024-04-26T01:07:53.417 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_no_require_squid": "false", 2024-04-26T01:07:53.417 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-26T01:07:53.417 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-26T01:07:53.417 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-26T01:07:53.417 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_down_mkfs_grace": "120", 2024-04-26T01:07:53.417 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_down_uptime_grace": "60", 2024-04-26T01:07:53.418 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_election_default_strategy": "1", 2024-04-26T01:07:53.418 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_election_timeout": "5.000000", 2024-04-26T01:07:53.418 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-26T01:07:53.418 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_elector_ping_divisor": "2", 2024-04-26T01:07:53.418 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-26T01:07:53.418 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_enable_op_tracker": "true", 2024-04-26T01:07:53.418 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_fake_pool_delete": "false", 2024-04-26T01:07:53.418 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_force_quorum_join": "false", 2024-04-26T01:07:53.418 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_globalid_prealloc": "10000", 2024-04-26T01:07:53.418 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_health_detail_to_clog": "false", 2024-04-26T01:07:53.418 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_health_log_update_period": "5", 2024-04-26T01:07:53.418 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_health_max_detail": "50", 2024-04-26T01:07:53.418 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_health_to_clog": "true", 2024-04-26T01:07:53.418 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_health_to_clog_interval": "600", 2024-04-26T01:07:53.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-26T01:07:53.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_host": "172.21.15.8,172.21.15.12,172.21.15.116", 2024-04-26T01:07:53.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_host_override": "", 2024-04-26T01:07:53.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_initial_members": "", 2024-04-26T01:07:53.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-26T01:07:53.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-26T01:07:53.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-26T01:07:53.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-26T01:07:53.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-26T01:07:53.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_lease": "5.000000", 2024-04-26T01:07:53.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-26T01:07:53.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-26T01:07:53.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_log_full_interval": "50", 2024-04-26T01:07:53.420 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_log_max": "10000", 2024-04-26T01:07:53.420 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_log_max_summary": "50", 2024-04-26T01:07:53.420 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-26T01:07:53.420 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_log_epochs": "500", 2024-04-26T01:07:53.420 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-26T01:07:53.420 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-26T01:07:53.420 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_osd": "10000", 2024-04-26T01:07:53.420 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_pg_per_osd": "10000", 2024-04-26T01:07:53.420 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_pool_pg_num": "65536", 2024-04-26T01:07:53.420 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-26T01:07:53.420 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-26T01:07:53.420 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mds_force_trim_to": "0", 2024-04-26T01:07:53.420 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mds_skip_sanity": "false", 2024-04-26T01:07:53.420 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_memory_autotune": "true", 2024-04-26T01:07:53.421 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_memory_target": "2147483648", 2024-04-26T01:07:53.421 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mgr_beacon_grace": "30", 2024-04-26T01:07:53.421 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-26T01:07:53.421 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mgr_digest_period": "5", 2024-04-26T01:07:53.421 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mgr_inactive_grace": "60", 2024-04-26T01:07:53.421 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-26T01:07:53.421 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-26T01:07:53.421 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_min_osdmap_epochs": "500", 2024-04-26T01:07:53.421 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_op_complaint_time": "30", 2024-04-26T01:07:53.421 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_op_history_duration": "600", 2024-04-26T01:07:53.421 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_op_history_size": "20", 2024-04-26T01:07:53.421 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_op_history_slow_op_size": "20", 2024-04-26T01:07:53.421 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-26T01:07:53.421 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_op_log_threshold": "5", 2024-04-26T01:07:53.422 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-26T01:07:53.422 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-26T01:07:53.422 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-26T01:07:53.422 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_auto_mark_in": "false", 2024-04-26T01:07:53.422 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-26T01:07:53.422 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-26T01:07:53.422 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-26T01:07:53.422 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_cache_size": "500", 2024-04-26T01:07:53.422 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-26T01:07:53.422 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-26T01:07:53.422 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-26T01:07:53.422 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_down_out_interval": "600", 2024-04-26T01:07:53.422 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-26T01:07:53.422 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-26T01:07:53.423 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_force_trim_to": "0", 2024-04-26T01:07:53.423 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-26T01:07:53.423 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-26T01:07:53.423 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-26T01:07:53.423 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-26T01:07:53.423 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-26T01:07:53.423 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-26T01:07:53.423 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_max_creating_pgs": "1024", 2024-04-26T01:07:53.423 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-26T01:07:53.423 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_min_down_reporters": "2", 2024-04-26T01:07:53.423 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-26T01:07:53.423 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-26T01:07:53.423 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-26T01:07:53.423 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-26T01:07:53.424 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-26T01:07:53.424 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-26T01:07:53.424 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_report_timeout": "900", 2024-04-26T01:07:53.424 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-26T01:07:53.424 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-26T01:07:53.424 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-26T01:07:53.424 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-26T01:07:53.424 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-26T01:07:53.424 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osdmap_full_prune_interval": "10", 2024-04-26T01:07:53.424 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osdmap_full_prune_min": "10000", 2024-04-26T01:07:53.424 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-04-26T01:07:53.424 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-26T01:07:53.424 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pg_stuck_threshold": "60", 2024-04-26T01:07:53.424 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-26T01:07:53.424 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-26T01:07:53.425 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-26T01:07:53.425 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-26T01:07:53.425 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-26T01:07:53.425 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-26T01:07:53.425 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_probe_timeout": "2.000000", 2024-04-26T01:07:53.425 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_reweight_max_change": "0.050000", 2024-04-26T01:07:53.425 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_reweight_max_osds": "4", 2024-04-26T01:07:53.425 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-26T01:07:53.425 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-26T01:07:53.425 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-26T01:07:53.425 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-26T01:07:53.425 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-26T01:07:53.425 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_scrub_interval": "300", 2024-04-26T01:07:53.425 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_scrub_max_keys": "100", 2024-04-26T01:07:53.425 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_scrub_timeout": "300", 2024-04-26T01:07:53.426 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_session_timeout": "300", 2024-04-26T01:07:53.426 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_smart_report_timeout": "5", 2024-04-26T01:07:53.426 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_stat_smooth_intervals": "6", 2024-04-26T01:07:53.426 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-26T01:07:53.426 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_stretch_pool_min_size": "2", 2024-04-26T01:07:53.426 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_stretch_pool_size": "4", 2024-04-26T01:07:53.426 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-26T01:07:53.426 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-26T01:07:53.426 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_sync_debug": "false", 2024-04-26T01:07:53.426 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-26T01:07:53.426 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-26T01:07:53.426 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_sync_provider_kill_at": "0", 2024-04-26T01:07:53.426 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_sync_requester_kill_at": "0", 2024-04-26T01:07:53.427 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_sync_timeout": "60.000000", 2024-04-26T01:07:53.427 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_target_pg_per_osd": "100", 2024-04-26T01:07:53.427 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_tick_interval": "5", 2024-04-26T01:07:53.427 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_timecheck_interval": "300.000000", 2024-04-26T01:07:53.427 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-26T01:07:53.427 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_use_min_delay_socket": "false", 2024-04-26T01:07:53.427 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_older_version_delay": "604800", 2024-04-26T01:07:53.427 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-26T01:07:53.427 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-26T01:07:53.427 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-26T01:07:53.427 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-26T01:07:53.427 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-26T01:07:53.427 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-26T01:07:53.427 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-26T01:07:53.428 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_misplaced": "false", 2024-04-26T01:07:53.428 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-26T01:07:53.428 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_older_version": "true", 2024-04-26T01:07:53.428 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-26T01:07:53.428 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-26T01:07:53.428 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-26T01:07:53.428 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-26T01:07:53.428 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-26T01:07:53.428 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-26T01:07:53.428 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-26T01:07:53.428 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-26T01:07:53.428 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-26T01:07:53.428 INFO:teuthology.orchestra.run.smithi008.stdout: "monmap": "", 2024-04-26T01:07:53.429 INFO:teuthology.orchestra.run.smithi008.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-26T01:07:53.429 INFO:teuthology.orchestra.run.smithi008.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-26T01:07:53.429 INFO:teuthology.orchestra.run.smithi008.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-26T01:07:53.429 INFO:teuthology.orchestra.run.smithi008.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-26T01:07:53.429 INFO:teuthology.orchestra.run.smithi008.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-26T01:07:53.429 INFO:teuthology.orchestra.run.smithi008.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-26T01:07:53.429 INFO:teuthology.orchestra.run.smithi008.stdout: "motr_tracing_enabled": "false", 2024-04-26T01:07:53.429 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_op_threads": "3", 2024-04-26T01:07:53.429 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-26T01:07:53.429 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_cm": "false", 2024-04-26T01:07:53.429 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_device_name": "", 2024-04-26T01:07:53.429 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_dscp": "96", 2024-04-26T01:07:53.429 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-26T01:07:53.429 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-26T01:07:53.430 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_local_gid": "", 2024-04-26T01:07:53.430 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-26T01:07:53.430 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_port_num": "1", 2024-04-26T01:07:53.430 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-26T01:07:53.430 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-26T01:07:53.430 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-26T01:07:53.430 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-26T01:07:53.430 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_sl": "3", 2024-04-26T01:07:53.430 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_support_srq": "true", 2024-04-26T01:07:53.430 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_type": "ib", 2024-04-26T01:07:53.430 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_reap_threshold": "5", 2024-04-26T01:07:53.430 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_before_connect": "false", 2024-04-26T01:07:53.431 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_ipv4": "true", 2024-04-26T01:07:53.431 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_ipv6": "false", 2024-04-26T01:07:53.431 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_msgr1": "true", 2024-04-26T01:07:53.431 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_msgr2": "true", 2024-04-26T01:07:53.431 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_port_max": "7568", 2024-04-26T01:07:53.431 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_port_min": "6800", 2024-04-26T01:07:53.431 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-26T01:07:53.431 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_retry_count": "3", 2024-04-26T01:07:53.431 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_retry_delay": "5", 2024-04-26T01:07:53.431 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_blackhole_client": "false", 2024-04-26T01:07:53.431 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_blackhole_mds": "false", 2024-04-26T01:07:53.431 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_blackhole_mgr": "false", 2024-04-26T01:07:53.431 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_blackhole_mon": "false", 2024-04-26T01:07:53.431 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_blackhole_osd": "false", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_client_mode": "crc secure", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_cluster_mode": "crc secure", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_cluster_type": "", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_compress_secure": "false", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_connection_idle_timeout": "900", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_connection_ready_timeout": "10", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_crc_data": "true", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_crc_header": "true", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_die_on_bad_msg": "false", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_die_on_bug": "true", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_die_on_old_message": "true", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_die_on_skipped_message": "false", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_coremask": "0xF", 2024-04-26T01:07:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-26T01:07:53.433 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-26T01:07:53.433 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_enable_tso": "true", 2024-04-26T01:07:53.433 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-26T01:07:53.433 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-26T01:07:53.433 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_hugepages": "", 2024-04-26T01:07:53.433 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-26T01:07:53.433 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-26T01:07:53.433 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_lro": "true", 2024-04-26T01:07:53.433 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_memory_channel": "4", 2024-04-26T01:07:53.433 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-26T01:07:53.433 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_pmd": "", 2024-04-26T01:07:53.433 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_port_id": "0", 2024-04-26T01:07:53.433 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-26T01:07:53.433 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-26T01:07:53.433 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dump_on_send": "false", 2024-04-26T01:07:53.434 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_initial_backoff": "0.200000", 2024-04-26T01:07:53.434 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_inject_delay_max": "1.000000", 2024-04-26T01:07:53.434 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_inject_delay_probability": "0.005000", 2024-04-26T01:07:53.434 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_inject_delay_type": "osd", 2024-04-26T01:07:53.434 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_inject_internal_delays": "0.002000", 2024-04-26T01:07:53.434 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_inject_network_congestion": "0", 2024-04-26T01:07:53.434 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_inject_socket_failures": "2500", 2024-04-26T01:07:53.434 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_learn_addr_from_peer": "true", 2024-04-26T01:07:53.434 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_max_accept_failures": "4", 2024-04-26T01:07:53.434 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_max_backoff": "15.000000", 2024-04-26T01:07:53.434 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_mon_client_mode": "secure crc", 2024-04-26T01:07:53.434 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-26T01:07:53.434 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_mon_service_mode": "secure crc", 2024-04-26T01:07:53.434 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_osd_compress_min_size": "1024", 2024-04-26T01:07:53.435 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_osd_compress_mode": "none", 2024-04-26T01:07:53.435 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-26T01:07:53.435 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-26T01:07:53.435 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_pq_min_cost": "65536", 2024-04-26T01:07:53.435 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_public_type": "", 2024-04-26T01:07:53.435 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_service_mode": "crc secure", 2024-04-26T01:07:53.435 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_tcp_listen_backlog": "512", 2024-04-26T01:07:53.435 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_tcp_nodelay": "true", 2024-04-26T01:07:53.435 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-26T01:07:53.435 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_tcp_rcvbuf": "0", 2024-04-26T01:07:53.435 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_type": "async+posix", 2024-04-26T01:07:53.435 INFO:teuthology.orchestra.run.smithi008.stdout: "no_config_file": "false", 2024-04-26T01:07:53.435 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_completion_locks_per_session": "32", 2024-04-26T01:07:53.435 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-26T01:07:53.435 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-26T01:07:53.436 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_inflight_ops": "1024", 2024-04-26T01:07:53.436 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-26T01:07:53.436 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-26T01:07:53.436 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_tick_interval": "5.000000", 2024-04-26T01:07:53.436 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_timeout": "10.000000", 2024-04-26T01:07:53.436 INFO:teuthology.orchestra.run.smithi008.stdout: "objectstore_blackhole": "false", 2024-04-26T01:07:53.436 INFO:teuthology.orchestra.run.smithi008.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-26T01:07:53.436 INFO:teuthology.orchestra.run.smithi008.stdout: "openssl_engine_opts": "", 2024-04-26T01:07:53.436 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_agent_delay_time": "5.000000", 2024-04-26T01:07:53.436 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_agent_hist_halflife": "1000", 2024-04-26T01:07:53.436 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_agent_max_low_ops": "2", 2024-04-26T01:07:53.436 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_agent_max_ops": "4", 2024-04-26T01:07:53.436 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-26T01:07:53.436 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-26T01:07:53.436 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_agent_slop": "0.020000", 2024-04-26T01:07:53.437 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-26T01:07:53.437 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-26T01:07:53.437 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_asio_thread_count": "2", 2024-04-26T01:07:53.437 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_async_recovery_min_cost": "100", 2024-04-26T01:07:53.437 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-26T01:07:53.437 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-26T01:07:53.437 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_backfill_scan_max": "512", 2024-04-26T01:07:53.437 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_backfill_scan_min": "64", 2024-04-26T01:07:53.437 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_backoff_on_degraded": "false", 2024-04-26T01:07:53.437 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_backoff_on_peering": "false", 2024-04-26T01:07:53.437 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_backoff_on_unfound": "true", 2024-04-26T01:07:53.437 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_beacon_report_interval": "300", 2024-04-26T01:07:53.437 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_bench_duration": "30", 2024-04-26T01:07:53.437 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-26T01:07:53.438 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_bench_max_block_size": "67108864", 2024-04-26T01:07:53.438 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-26T01:07:53.438 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_blkin_trace_all": "false", 2024-04-26T01:07:53.438 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-26T01:07:53.438 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-26T01:07:53.438 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-26T01:07:53.438 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-26T01:07:53.438 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_check_for_log_corruption": "false", 2024-04-26T01:07:53.438 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-26T01:07:53.438 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-26T01:07:53.438 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-04-26T01:07:53.438 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-26T01:07:53.438 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_class_update_on_start": "true", 2024-04-26T01:07:53.438 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_client_message_cap": "256", 2024-04-26T01:07:53.439 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_client_message_size_cap": "524288000", 2024-04-26T01:07:53.439 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_client_op_priority": "63", 2024-04-26T01:07:53.439 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_client_watch_timeout": "30", 2024-04-26T01:07:53.439 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_command_max_records": "256", 2024-04-26T01:07:53.439 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_compact_on_start": "false", 2024-04-26T01:07:53.439 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-26T01:07:53.439 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-26T01:07:53.439 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-26T01:07:53.439 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_crush_update_on_start": "true", 2024-04-26T01:07:53.439 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_crush_update_weight_set": "true", 2024-04-26T01:07:53.439 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-26T01:07:53.439 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-26T01:07:53.439 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-26T01:07:53.439 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-26T01:07:53.439 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-26T01:07:53.440 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_feed_pullee": "-1", 2024-04-26T01:07:53.440 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-26T01:07:53.440 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-26T01:07:53.440 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-26T01:07:53.440 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_misdirected_ops": "false", 2024-04-26T01:07:53.440 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_no_acting_change": "false", 2024-04-26T01:07:53.440 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_no_purge_strays": "false", 2024-04-26T01:07:53.440 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_op_order": "false", 2024-04-26T01:07:53.440 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-26T01:07:53.440 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-26T01:07:53.440 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-26T01:07:53.440 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-26T01:07:53.440 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_shutdown": "false", 2024-04-26T01:07:53.440 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-26T01:07:53.440 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-26T01:07:53.441 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_trim_objects": "false", 2024-04-26T01:07:53.441 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-26T01:07:53.441 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-26T01:07:53.441 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_verify_snaps": "false", 2024-04-26T01:07:53.441 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-26T01:07:53.441 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-26T01:07:53.441 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_deep_scrub_keys": "1024", 2024-04-26T01:07:53.441 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-26T01:07:53.441 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-26T01:07:53.441 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-26T01:07:53.441 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_deep_scrub_stride": "524288", 2024-04-26T01:07:53.442 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-26T01:07:53.442 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-26T01:07:53.442 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_default_notify_timeout": "30", 2024-04-26T01:07:53.442 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_delete_sleep": "0.000000", 2024-04-26T01:07:53.442 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-26T01:07:53.442 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-26T01:07:53.442 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-26T01:07:53.442 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_discard_disconnected_ops": "true", 2024-04-26T01:07:53.442 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_enable_op_tracker": "true", 2024-04-26T01:07:53.442 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-26T01:07:53.442 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-26T01:07:53.442 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-26T01:07:53.442 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-26T01:07:53.443 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_fast_info": "true", 2024-04-26T01:07:53.443 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_fast_shutdown": "true", 2024-04-26T01:07:53.443 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-26T01:07:53.443 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-26T01:07:53.443 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-26T01:07:53.443 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-26T01:07:53.443 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-26T01:07:53.443 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_function_tracing": "false", 2024-04-26T01:07:53.443 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_heartbeat_grace": "20", 2024-04-26T01:07:53.443 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_heartbeat_interval": "6", 2024-04-26T01:07:53.443 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-26T01:07:53.443 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_heartbeat_min_peers": "10", 2024-04-26T01:07:53.443 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_heartbeat_min_size": "2000", 2024-04-26T01:07:53.443 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_heartbeat_stale": "600", 2024-04-26T01:07:53.444 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-26T01:07:53.444 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_hit_set_max_size": "100000", 2024-04-26T01:07:53.444 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_hit_set_min_size": "1000", 2024-04-26T01:07:53.444 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-26T01:07:53.444 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-26T01:07:53.444 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-26T01:07:53.444 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-26T01:07:53.444 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-26T01:07:53.444 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-26T01:07:53.444 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_journal_size": "5120", 2024-04-26T01:07:53.444 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_kill_backfill_at": "0", 2024-04-26T01:07:53.444 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-26T01:07:53.444 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_map_cache_size": "50", 2024-04-26T01:07:53.445 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_map_dedup": "true", 2024-04-26T01:07:53.445 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_map_message_max": "40", 2024-04-26T01:07:53.445 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-26T01:07:53.445 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_map_share_max_epochs": "40", 2024-04-26T01:07:53.445 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_attr_name_len": "100", 2024-04-26T01:07:53.445 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_attr_size": "0", 2024-04-26T01:07:53.445 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_backfills": "1", 2024-04-26T01:07:53.445 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_markdown_count": "5", 2024-04-26T01:07:53.445 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_markdown_period": "600", 2024-04-26T01:07:53.445 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_object_name_len": "2048", 2024-04-26T01:07:53.445 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_object_namespace_len": "256", 2024-04-26T01:07:53.445 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_object_size": "134217728", 2024-04-26T01:07:53.445 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-26T01:07:53.445 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-26T01:07:53.446 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_pg_blocked_by": "16", 2024-04-26T01:07:53.446 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_pg_log_entries": "10000", 2024-04-26T01:07:53.446 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-26T01:07:53.446 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_pgls": "1024", 2024-04-26T01:07:53.446 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_push_cost": "8388608", 2024-04-26T01:07:53.446 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_push_objects": "10", 2024-04-26T01:07:53.446 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_scrubs": "3", 2024-04-26T01:07:53.446 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-26T01:07:53.446 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_trimming_pgs": "2", 2024-04-26T01:07:53.446 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_write_op_reply_len": "64", 2024-04-26T01:07:53.446 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_write_size": "90", 2024-04-26T01:07:53.447 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-26T01:07:53.447 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-26T01:07:53.447 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-26T01:07:53.447 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-26T01:07:53.447 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-26T01:07:53.447 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-26T01:07:53.447 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-26T01:07:53.447 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-26T01:07:53.447 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_profile": "balanced", 2024-04-26T01:07:53.447 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-26T01:07:53.447 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-26T01:07:53.447 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-26T01:07:53.447 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-26T01:07:53.447 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-26T01:07:53.448 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-26T01:07:53.448 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-26T01:07:53.448 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-26T01:07:53.448 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-26T01:07:53.448 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-26T01:07:53.448 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-26T01:07:53.448 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_memory_base": "805306368", 2024-04-26T01:07:53.448 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_memory_cache_min": "134217728", 2024-04-26T01:07:53.448 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-26T01:07:53.448 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-26T01:07:53.448 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_memory_target": "4294967296", 2024-04-26T01:07:53.448 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_memory_target_autotune": "false", 2024-04-26T01:07:53.448 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-26T01:07:53.448 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_min_pg_log_entries": "250", 2024-04-26T01:07:53.449 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_min_recovery_priority": "0", 2024-04-26T01:07:53.449 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-26T01:07:53.449 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-26T01:07:53.449 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mon_report_interval": "5", 2024-04-26T01:07:53.449 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-26T01:07:53.449 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-26T01:07:53.449 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_num_cache_shards": "32", 2024-04-26T01:07:53.449 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_num_op_tracker_shard": "32", 2024-04-26T01:07:53.449 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_numa_auto_affinity": "true", 2024-04-26T01:07:53.449 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_numa_node": "-1", 2024-04-26T01:07:53.449 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_numa_prefer_iface": "true", 2024-04-26T01:07:53.449 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-04-26T01:07:53.449 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_objecter_finishers": "1", 2024-04-26T01:07:53.449 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_objectstore": "bluestore", 2024-04-26T01:07:53.450 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_objectstore_fuse": "false", 2024-04-26T01:07:53.450 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_objectstore_tracing": "false", 2024-04-26T01:07:53.450 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_complaint_time": "30.000000", 2024-04-26T01:07:53.450 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_history_duration": "600", 2024-04-26T01:07:53.450 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_history_size": "20", 2024-04-26T01:07:53.450 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_history_slow_op_size": "20", 2024-04-26T01:07:53.450 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-26T01:07:53.450 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_log_threshold": "5", 2024-04-26T01:07:53.450 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_num_shards": "0", 2024-04-26T01:07:53.450 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_num_shards_hdd": "5", 2024-04-26T01:07:53.450 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_num_shards_ssd": "8", 2024-04-26T01:07:53.450 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-26T01:07:53.450 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-26T01:07:53.451 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-26T01:07:53.451 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-26T01:07:53.451 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_pq_min_cost": "65536", 2024-04-26T01:07:53.451 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-26T01:07:53.451 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_queue_cut_off": "high", 2024-04-26T01:07:53.451 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-26T01:07:53.451 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_thread_timeout": "15", 2024-04-26T01:07:53.451 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_open_classes_on_start": "true", 2024-04-26T01:07:53.451 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_os_flags": "0", 2024-04-26T01:07:53.451 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_peering_op_priority": "255", 2024-04-26T01:07:53.451 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_delete_cost": "1048576", 2024-04-26T01:07:53.451 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_delete_priority": "5", 2024-04-26T01:07:53.451 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-26T01:07:53.452 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-26T01:07:53.452 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-26T01:07:53.452 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_log_trim_max": "10000", 2024-04-26T01:07:53.452 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_log_trim_min": "100", 2024-04-26T01:07:53.452 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-26T01:07:53.452 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-26T01:07:53.452 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-26T01:07:53.452 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-26T01:07:53.452 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-26T01:07:53.452 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-26T01:07:53.452 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-26T01:07:53.452 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-26T01:07:53.452 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-26T01:07:53.452 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-26T01:07:53.453 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_crimson": "false", 2024-04-26T01:07:53.453 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-26T01:07:53.453 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-26T01:07:53.453 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-26T01:07:53.453 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-26T01:07:53.453 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-26T01:07:53.453 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-26T01:07:53.453 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-26T01:07:53.453 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-26T01:07:53.453 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_flags": "0", 2024-04-26T01:07:53.453 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-26T01:07:53.453 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_min_size": "0", 2024-04-26T01:07:53.453 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-26T01:07:53.453 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_pg_num": "32", 2024-04-26T01:07:53.454 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_pgp_num": "0", 2024-04-26T01:07:53.454 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-26T01:07:53.454 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_read_ratio": "70", 2024-04-26T01:07:53.454 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_size": "2", 2024-04-26T01:07:53.454 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_type": "replicated", 2024-04-26T01:07:53.454 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-26T01:07:53.454 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-26T01:07:53.454 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_push_per_object_cost": "1000", 2024-04-26T01:07:53.454 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-26T01:07:53.454 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recover_clone_overlap": "true", 2024-04-26T01:07:53.454 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-26T01:07:53.454 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_cost": "20971520", 2024-04-26T01:07:53.455 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-26T01:07:53.455 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_max_active": "0", 2024-04-26T01:07:53.455 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-26T01:07:53.455 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-26T01:07:53.455 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-26T01:07:53.455 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-26T01:07:53.455 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_max_single_start": "1", 2024-04-26T01:07:53.455 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_op_priority": "3", 2024-04-26T01:07:53.455 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-26T01:07:53.455 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_priority": "5", 2024-04-26T01:07:53.455 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-26T01:07:53.455 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_sleep": "0.000000", 2024-04-26T01:07:53.455 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-26T01:07:53.456 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-26T01:07:53.456 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-26T01:07:53.456 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_repair_during_recovery": "false", 2024-04-26T01:07:53.456 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_requested_scrub_priority": "120", 2024-04-26T01:07:53.456 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-26T01:07:53.456 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-26T01:07:53.456 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_auto_repair": "false", 2024-04-26T01:07:53.456 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-26T01:07:53.456 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-26T01:07:53.456 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_begin_hour": "0", 2024-04-26T01:07:53.456 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_begin_week_day": "0", 2024-04-26T01:07:53.456 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_chunk_max": "25", 2024-04-26T01:07:53.456 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_chunk_min": "5", 2024-04-26T01:07:53.456 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_cost": "52428800", 2024-04-26T01:07:53.457 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_disable_reservation_queuing": "true", 2024-04-26T01:07:53.457 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_during_recovery": "false", 2024-04-26T01:07:53.457 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_end_hour": "0", 2024-04-26T01:07:53.457 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_end_week_day": "0", 2024-04-26T01:07:53.457 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_event_cost": "4096", 2024-04-26T01:07:53.457 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-26T01:07:53.457 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-26T01:07:53.457 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_invalid_stats": "true", 2024-04-26T01:07:53.457 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-26T01:07:53.457 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-26T01:07:53.457 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_max_preemptions": "5", 2024-04-26T01:07:53.457 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-26T01:07:53.457 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_priority": "5", 2024-04-26T01:07:53.457 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-26T01:07:53.458 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_sleep": "0.000000", 2024-04-26T01:07:53.458 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-26T01:07:53.458 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-26T01:07:53.458 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-26T01:07:53.458 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_skip_data_digest": "false", 2024-04-26T01:07:53.458 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_smart_report_timeout": "5", 2024-04-26T01:07:53.458 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_snap_trim_cost": "1048576", 2024-04-26T01:07:53.458 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_snap_trim_priority": "5", 2024-04-26T01:07:53.458 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-26T01:07:53.458 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-26T01:07:53.458 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-26T01:07:53.458 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-26T01:07:53.458 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-26T01:07:53.459 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-26T01:07:53.459 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-26T01:07:53.459 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_target_transaction_size": "30", 2024-04-26T01:07:53.459 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-26T01:07:53.459 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-26T01:07:53.459 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-26T01:07:53.459 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-26T01:07:53.459 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-26T01:07:53.459 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-26T01:07:53.459 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-26T01:07:53.459 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-26T01:07:53.459 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-26T01:07:53.459 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-26T01:07:53.460 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tracing": "false", 2024-04-26T01:07:53.460 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_use_stale_snap": "false", 2024-04-26T01:07:53.460 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-26T01:07:53.460 INFO:teuthology.orchestra.run.smithi008.stdout: "osdc_blkin_trace_all": "false", 2024-04-26T01:07:53.460 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_kill_at": "0", 2024-04-26T01:07:53.460 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_max_join_drift": "10", 2024-04-26T01:07:53.460 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_min": "500", 2024-04-26T01:07:53.460 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_min_wait": "0.050000", 2024-04-26T01:07:53.460 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_propose_interval": "1.000000", 2024-04-26T01:07:53.460 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_service_trim_max": "500", 2024-04-26T01:07:53.460 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-26T01:07:53.460 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_service_trim_min": "250", 2024-04-26T01:07:53.460 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_stash_full_interval": "25", 2024-04-26T01:07:53.460 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_trim_max": "500", 2024-04-26T01:07:53.461 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_trim_min": "250", 2024-04-26T01:07:53.461 INFO:teuthology.orchestra.run.smithi008.stdout: "perf": "true", 2024-04-26T01:07:53.461 INFO:teuthology.orchestra.run.smithi008.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-26T01:07:53.461 INFO:teuthology.orchestra.run.smithi008.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-26T01:07:53.461 INFO:teuthology.orchestra.run.smithi008.stdout: "plugin_dir": "/usr/lib/ceph", 2024-04-26T01:07:53.461 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "-", 2024-04-26T01:07:53.461 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrv": "", 2024-04-26T01:07:53.461 INFO:teuthology.orchestra.run.smithi008.stdout: "public_bind_addr": "-", 2024-04-26T01:07:53.461 INFO:teuthology.orchestra.run.smithi008.stdout: "public_network": "", 2024-04-26T01:07:53.461 INFO:teuthology.orchestra.run.smithi008.stdout: "public_network_interface": "", 2024-04-26T01:07:53.461 INFO:teuthology.orchestra.run.smithi008.stdout: "qat_compressor_enabled": "false", 2024-04-26T01:07:53.461 INFO:teuthology.orchestra.run.smithi008.stdout: "qat_compressor_session_max_number": "256", 2024-04-26T01:07:53.461 INFO:teuthology.orchestra.run.smithi008.stdout: "rados_mon_op_timeout": "0", 2024-04-26T01:07:53.461 INFO:teuthology.orchestra.run.smithi008.stdout: "rados_osd_op_timeout": "0", 2024-04-26T01:07:53.462 INFO:teuthology.orchestra.run.smithi008.stdout: "rados_tracing": "false", 2024-04-26T01:07:53.462 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_atime_update_interval": "60", 2024-04-26T01:07:53.462 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-26T01:07:53.462 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_balance_parent_reads": "false", 2024-04-26T01:07:53.462 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_balance_snap_reads": "false", 2024-04-26T01:07:53.462 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_blkin_trace_all": "false", 2024-04-26T01:07:53.462 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-26T01:07:53.462 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-26T01:07:53.462 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache": "true", 2024-04-26T01:07:53.462 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-26T01:07:53.462 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-26T01:07:53.462 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-26T01:07:53.462 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-26T01:07:53.463 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_policy": "writearound", 2024-04-26T01:07:53.463 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_size": "33554432", 2024-04-26T01:07:53.463 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-26T01:07:53.463 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-26T01:07:53.463 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_clone_copy_on_read": "false", 2024-04-26T01:07:53.463 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_compression_hint": "none", 2024-04-26T01:07:53.463 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_concurrent_management_ops": "10", 2024-04-26T01:07:53.463 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-26T01:07:53.463 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_clone_format": "auto", 2024-04-26T01:07:53.463 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_data_pool": "", 2024-04-26T01:07:53.463 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_features": "61", 2024-04-26T01:07:53.463 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_format": "2", 2024-04-26T01:07:53.463 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_map_options": "", 2024-04-26T01:07:53.463 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_order": "22", 2024-04-26T01:07:53.464 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_pool": "rbd", 2024-04-26T01:07:53.464 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-26T01:07:53.464 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_stripe_count": "0", 2024-04-26T01:07:53.464 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_stripe_unit": "0", 2024-04-26T01:07:53.464 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-26T01:07:53.464 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-26T01:07:53.464 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-26T01:07:53.464 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_enable_alloc_hint": "true", 2024-04-26T01:07:53.464 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-26T01:07:53.464 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_io_scheduler": "simple", 2024-04-26T01:07:53.464 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-26T01:07:53.464 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-26T01:07:53.464 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-26T01:07:53.464 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-26T01:07:53.465 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-26T01:07:53.465 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-26T01:07:53.465 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-26T01:07:53.465 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-26T01:07:53.465 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-26T01:07:53.465 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_order": "24", 2024-04-26T01:07:53.465 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_pool": "", 2024-04-26T01:07:53.465 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_splay_width": "4", 2024-04-26T01:07:53.465 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_localize_parent_reads": "false", 2024-04-26T01:07:53.465 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_localize_snap_reads": "false", 2024-04-26T01:07:53.465 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-26T01:07:53.465 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-26T01:07:53.466 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-26T01:07:53.466 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-26T01:07:53.466 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-26T01:07:53.466 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-26T01:07:53.466 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-26T01:07:53.466 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-26T01:07:53.466 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-26T01:07:53.466 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-26T01:07:53.466 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-26T01:07:53.466 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-26T01:07:53.466 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-26T01:07:53.466 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-26T01:07:53.467 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-26T01:07:53.467 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-26T01:07:53.467 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-26T01:07:53.467 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-26T01:07:53.467 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-26T01:07:53.467 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-26T01:07:53.467 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-26T01:07:53.467 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-26T01:07:53.467 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-26T01:07:53.467 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-26T01:07:53.467 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-26T01:07:53.467 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-26T01:07:53.467 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-26T01:07:53.467 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-26T01:07:53.468 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-26T01:07:53.468 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-26T01:07:53.468 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-26T01:07:53.468 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mtime_update_interval": "60", 2024-04-26T01:07:53.468 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_non_blocking_aio": "true", 2024-04-26T01:07:53.468 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_op_thread_timeout": "60", 2024-04-26T01:07:53.468 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_op_threads": "1", 2024-04-26T01:07:53.468 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_parent_cache_enabled": "false", 2024-04-26T01:07:53.468 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-26T01:07:53.468 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-26T01:07:53.468 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-26T01:07:53.468 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_plugins": "", 2024-04-26T01:07:53.468 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_bps_burst": "0", 2024-04-26T01:07:53.468 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-26T01:07:53.469 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_bps_limit": "0", 2024-04-26T01:07:53.469 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_exclude_ops": "0", 2024-04-26T01:07:53.469 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_iops_burst": "0", 2024-04-26T01:07:53.469 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-26T01:07:53.469 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_iops_limit": "0", 2024-04-26T01:07:53.469 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-26T01:07:53.469 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-26T01:07:53.469 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-26T01:07:53.469 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-26T01:07:53.469 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-26T01:07:53.469 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-26T01:07:53.469 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-26T01:07:53.469 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-26T01:07:53.470 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-26T01:07:53.470 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-26T01:07:53.470 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-26T01:07:53.470 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-26T01:07:53.470 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-26T01:07:53.470 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-26T01:07:53.470 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_read_from_replica_policy": "default", 2024-04-26T01:07:53.470 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-26T01:07:53.470 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-26T01:07:53.470 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-26T01:07:53.470 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-26T01:07:53.470 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_skip_partial_discard": "true", 2024-04-26T01:07:53.470 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-26T01:07:53.470 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_tracing": "false", 2024-04-26T01:07:53.471 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_validate_names": "true", 2024-04-26T01:07:53.471 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_validate_pool": "true", 2024-04-26T01:07:53.471 INFO:teuthology.orchestra.run.smithi008.stdout: "restapi_base_url": "", 2024-04-26T01:07:53.471 INFO:teuthology.orchestra.run.smithi008.stdout: "restapi_log_level": "", 2024-04-26T01:07:53.471 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_acl_grants_max_num": "100", 2024-04-26T01:07:53.471 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_admin_entry": "admin", 2024-04-26T01:07:53.471 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-26T01:07:53.471 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_backend_store": "rados", 2024-04-26T01:07:53.471 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_barbican_url": "", 2024-04-26T01:07:53.471 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_beast_enable_async": "true", 2024-04-26T01:07:53.471 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_counters_cache": "false", 2024-04-26T01:07:53.471 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-26T01:07:53.471 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-26T01:07:53.471 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-26T01:07:53.472 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-26T01:07:53.472 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-26T01:07:53.472 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-26T01:07:53.472 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-26T01:07:53.472 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-26T01:07:53.472 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_cache_enabled": "true", 2024-04-26T01:07:53.472 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_cache_expiry_interval": "900", 2024-04-26T01:07:53.472 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_cache_lru_size": "10000", 2024-04-26T01:07:53.472 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_config_store": "rados", 2024-04-26T01:07:53.472 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_content_length_compat": "false", 2024-04-26T01:07:53.472 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_copy_obj_progress": "true", 2024-04-26T01:07:53.472 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-26T01:07:53.472 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_cors_rules_max_num": "100", 2024-04-26T01:07:53.472 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_cross_domain_policy": "", 2024-04-26T01:07:53.473 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-26T01:07:53.473 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_addr": "", 2024-04-26T01:07:53.473 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-26T01:07:53.473 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-26T01:07:53.473 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-26T01:07:53.473 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-26T01:07:53.473 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_password": "", 2024-04-26T01:07:53.473 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-26T01:07:53.473 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_username": "", 2024-04-26T01:07:53.473 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_require_ssl": "true", 2024-04-26T01:07:53.473 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-26T01:07:53.473 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-26T01:07:53.473 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-26T01:07:53.474 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-26T01:07:53.474 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-26T01:07:53.474 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-26T01:07:53.474 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-26T01:07:53.474 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-26T01:07:53.474 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-26T01:07:53.474 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-26T01:07:53.474 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-26T01:07:53.474 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-26T01:07:53.474 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-26T01:07:53.474 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-26T01:07:53.474 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-26T01:07:53.474 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_addr": "", 2024-04-26T01:07:53.474 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_auth": "token", 2024-04-26T01:07:53.475 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_namespace": "", 2024-04-26T01:07:53.475 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_prefix": "", 2024-04-26T01:07:53.475 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-26T01:07:53.475 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-26T01:07:53.475 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-26T01:07:53.475 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-26T01:07:53.475 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_token_file": "", 2024-04-26T01:07:53.475 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-26T01:07:53.475 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_curl_buffersize": "524288", 2024-04-26T01:07:53.475 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-26T01:07:53.475 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_curl_low_speed_time": "300", 2024-04-26T01:07:53.475 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-26T01:07:53.475 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-26T01:07:53.475 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-26T01:07:53.476 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-26T01:07:53.476 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-26T01:07:53.476 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-26T01:07:53.476 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-26T01:07:53.476 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-26T01:07:53.476 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-26T01:07:53.476 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-26T01:07:53.476 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-26T01:07:53.476 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d4n_port": "6379", 2024-04-26T01:07:53.476 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-26T01:07:53.476 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data_log_changes_size": "1000", 2024-04-26T01:07:53.476 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data_log_num_shards": "128", 2024-04-26T01:07:53.476 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-26T01:07:53.477 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data_log_window": "30", 2024-04-26T01:07:53.477 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-26T01:07:53.477 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-26T01:07:53.477 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-26T01:07:53.477 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-26T01:07:53.477 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-26T01:07:53.477 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-26T01:07:53.477 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-26T01:07:53.477 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-26T01:07:53.477 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-26T01:07:53.477 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-26T01:07:53.477 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-26T01:07:53.477 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-26T01:07:53.477 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-26T01:07:53.478 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-26T01:07:53.478 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-26T01:07:53.478 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-26T01:07:53.478 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-26T01:07:53.478 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-26T01:07:53.478 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-26T01:07:53.478 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-26T01:07:53.478 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-26T01:07:53.478 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-26T01:07:53.478 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-26T01:07:53.478 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-26T01:07:53.478 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-26T01:07:53.478 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dns_name": "", 2024-04-26T01:07:53.478 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dns_s3website_name": "", 2024-04-26T01:07:53.479 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dynamic_resharding": "true", 2024-04-26T01:07:53.479 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-26T01:07:53.479 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enable_gc_threads": "true", 2024-04-26T01:07:53.479 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enable_lc_threads": "true", 2024-04-26T01:07:53.479 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enable_ops_log": "false", 2024-04-26T01:07:53.479 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enable_quota_threads": "true", 2024-04-26T01:07:53.479 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enable_static_website": "false", 2024-04-26T01:07:53.479 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enable_usage_log": "false", 2024-04-26T01:07:53.479 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enforce_swift_acls": "true", 2024-04-26T01:07:53.479 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_exit_timeout_secs": "120", 2024-04-26T01:07:53.479 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_expose_bucket": "false", 2024-04-26T01:07:53.479 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_extended_http_attrs": "", 2024-04-26T01:07:53.480 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_filter": "none", 2024-04-26T01:07:53.480 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-26T01:07:53.480 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_frontends": "beast port=7480", 2024-04-26T01:07:53.480 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-26T01:07:53.480 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_max_deferred": "50", 2024-04-26T01:07:53.480 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-26T01:07:53.480 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_max_objs": "32", 2024-04-26T01:07:53.480 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-26T01:07:53.480 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-26T01:07:53.480 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-26T01:07:53.480 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-26T01:07:53.480 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_processor_period": "3600", 2024-04-26T01:07:53.480 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-26T01:07:53.480 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-26T01:07:53.481 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-26T01:07:53.481 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-26T01:07:53.481 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_init_timeout": "300", 2024-04-26T01:07:53.481 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-26T01:07:53.481 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-26T01:07:53.481 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_kafka_connection_idle": "300", 2024-04-26T01:07:53.481 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-26T01:07:53.481 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-26T01:07:53.481 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-26T01:07:53.481 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-26T01:07:53.481 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-26T01:07:53.481 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_domain": "", 2024-04-26T01:07:53.481 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_password": "", 2024-04-26T01:07:53.481 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_password_path": "", 2024-04-26T01:07:53.482 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_project": "", 2024-04-26T01:07:53.482 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_tenant": "", 2024-04-26T01:07:53.482 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_token": "", 2024-04-26T01:07:53.482 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_token_path": "", 2024-04-26T01:07:53.482 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_user": "", 2024-04-26T01:07:53.482 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_api_version": "2", 2024-04-26T01:07:53.482 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_barbican_domain": "", 2024-04-26T01:07:53.482 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_barbican_password": "", 2024-04-26T01:07:53.482 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_barbican_project": "", 2024-04-26T01:07:53.482 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-26T01:07:53.482 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_barbican_user": "", 2024-04-26T01:07:53.482 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-26T01:07:53.482 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-26T01:07:53.482 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-26T01:07:53.483 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-26T01:07:53.483 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-26T01:07:53.483 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_url": "", 2024-04-26T01:07:53.483 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-26T01:07:53.483 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lc_debug_interval": "-1", 2024-04-26T01:07:53.483 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lc_lock_max_time": "90", 2024-04-26T01:07:53.483 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lc_max_objs": "32", 2024-04-26T01:07:53.483 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lc_max_rules": "1000", 2024-04-26T01:07:53.483 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lc_max_worker": "3", 2024-04-26T01:07:53.483 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-26T01:07:53.483 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lc_thread_delay": "0", 2024-04-26T01:07:53.483 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-26T01:07:53.484 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ldap_dnattr": "uid", 2024-04-26T01:07:53.484 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-26T01:07:53.484 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ldap_searchfilter": "", 2024-04-26T01:07:53.484 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-26T01:07:53.484 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ldap_uri": "", 2024-04-26T01:07:53.484 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-26T01:07:53.484 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-26T01:07:53.484 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-26T01:07:53.484 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_log_http_headers": "", 2024-04-26T01:07:53.484 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-26T01:07:53.484 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-26T01:07:53.484 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_log_object_name_utc": "false", 2024-04-26T01:07:53.485 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-26T01:07:53.485 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-26T01:07:53.485 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_attr_name_len": "0", 2024-04-26T01:07:53.485 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_attr_size": "0", 2024-04-26T01:07:53.485 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-26T01:07:53.485 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_chunk_size": "4194304", 2024-04-26T01:07:53.485 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-26T01:07:53.485 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-26T01:07:53.485 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-26T01:07:53.485 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_listing_results": "1000", 2024-04-26T01:07:53.485 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_notify_retries": "10", 2024-04-26T01:07:53.485 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-26T01:07:53.485 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_put_param_size": "1048576", 2024-04-26T01:07:53.485 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_put_size": "5368709120", 2024-04-26T01:07:53.486 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_slo_entries": "1000", 2024-04-26T01:07:53.486 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_md_log_max_shards": "64", 2024-04-26T01:07:53.486 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-26T01:07:53.486 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-26T01:07:53.486 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-26T01:07:53.486 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-26T01:07:53.486 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_mp_lock_max_time": "600", 2024-04-26T01:07:53.486 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-26T01:07:53.486 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-26T01:07:53.486 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-26T01:07:53.486 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-26T01:07:53.486 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-26T01:07:53.486 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-26T01:07:53.486 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-26T01:07:53.487 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-26T01:07:53.487 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_max_gc": "300", 2024-04-26T01:07:53.487 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-26T01:07:53.487 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-26T01:07:53.487 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-26T01:07:53.487 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-26T01:07:53.487 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-26T01:07:53.487 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-26T01:07:53.487 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-26T01:07:53.487 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_num_async_rados_threads": "32", 2024-04-26T01:07:53.487 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_num_control_oids": "8", 2024-04-26T01:07:53.487 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_numa_node": "-1", 2024-04-26T01:07:53.487 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-26T01:07:53.487 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-26T01:07:53.488 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_objexp_chunk_size": "100", 2024-04-26T01:07:53.488 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_objexp_gc_interval": "600", 2024-04-26T01:07:53.488 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-26T01:07:53.488 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-26T01:07:53.488 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-26T01:07:53.488 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_op_thread_timeout": "600", 2024-04-26T01:07:53.488 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_op_tracing": "false", 2024-04-26T01:07:53.488 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_opa_token": "", 2024-04-26T01:07:53.488 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_opa_url": "", 2024-04-26T01:07:53.488 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_opa_verify_ssl": "true", 2024-04-26T01:07:53.488 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-26T01:07:53.488 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-26T01:07:53.488 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ops_log_rados": "false", 2024-04-26T01:07:53.488 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ops_log_socket_path": "", 2024-04-26T01:07:53.489 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-26T01:07:53.489 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-26T01:07:53.489 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-26T01:07:53.489 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_period_push_interval": "2.000000", 2024-04-26T01:07:53.489 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-26T01:07:53.489 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-26T01:07:53.489 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-26T01:07:53.489 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-26T01:07:53.489 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_posix_cache_lanes": "3", 2024-04-26T01:07:53.489 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-26T01:07:53.489 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-26T01:07:53.489 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_posix_cache_partitions": "3", 2024-04-26T01:07:53.489 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-26T01:07:53.489 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_print_continue": "true", 2024-04-26T01:07:53.489 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-26T01:07:53.490 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-26T01:07:53.490 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-26T01:07:53.490 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-26T01:07:53.490 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-26T01:07:53.490 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_rados_tracing": "false", 2024-04-26T01:07:53.490 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_realm": "", 2024-04-26T01:07:53.490 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_realm_id": "", 2024-04-26T01:07:53.490 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-26T01:07:53.490 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_region": "", 2024-04-26T01:07:53.490 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-26T01:07:53.490 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-26T01:07:53.490 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-26T01:07:53.490 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-26T01:07:53.491 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_request_uri": "", 2024-04-26T01:07:53.491 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_reshard_batch_size": "64", 2024-04-26T01:07:53.491 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-26T01:07:53.491 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_reshard_max_aio": "128", 2024-04-26T01:07:53.491 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_reshard_num_logs": "16", 2024-04-26T01:07:53.491 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_reshard_thread_interval": "600", 2024-04-26T01:07:53.491 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_resolve_cname": "false", 2024-04-26T01:07:53.491 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-26T01:07:53.491 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_run_sync_thread": "true", 2024-04-26T01:07:53.491 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-26T01:07:53.491 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-26T01:07:53.491 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-26T01:07:53.491 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-26T01:07:53.491 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-26T01:07:53.491 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-26T01:07:53.492 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-26T01:07:53.492 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-26T01:07:53.492 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-26T01:07:53.492 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_scheduler_type": "throttler", 2024-04-26T01:07:53.492 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_script_uri": "", 2024-04-26T01:07:53.492 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_service_provider_name": "", 2024-04-26T01:07:53.492 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-26T01:07:53.492 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sts_client_id": "", 2024-04-26T01:07:53.492 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sts_client_secret": "", 2024-04-26T01:07:53.492 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sts_entry": "sts", 2024-04-26T01:07:53.492 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sts_key": "sts", 2024-04-26T01:07:53.492 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-26T01:07:53.492 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sts_min_session_duration": "900", 2024-04-26T01:07:53.492 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sts_token_introspection_url": "", 2024-04-26T01:07:53.493 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_account_in_url": "false", 2024-04-26T01:07:53.493 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_auth_entry": "auth", 2024-04-26T01:07:53.493 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_auth_url": "", 2024-04-26T01:07:53.493 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_custom_header": "", 2024-04-26T01:07:53.493 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-26T01:07:53.493 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_need_stats": "true", 2024-04-26T01:07:53.493 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_tenant_name": "", 2024-04-26T01:07:53.493 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_token_expiration": "86400", 2024-04-26T01:07:53.493 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_url": "", 2024-04-26T01:07:53.493 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_url_prefix": "swift", 2024-04-26T01:07:53.493 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-26T01:07:53.493 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-26T01:07:53.493 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-26T01:07:53.493 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_lease_period": "120", 2024-04-26T01:07:53.493 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-26T01:07:53.494 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-26T01:07:53.494 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-26T01:07:53.494 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-26T01:07:53.494 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-26T01:07:53.494 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-26T01:07:53.494 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-26T01:07:53.494 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-26T01:07:53.494 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-26T01:07:53.494 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_thread_pool_size": "512", 2024-04-26T01:07:53.494 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-26T01:07:53.494 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-26T01:07:53.494 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-26T01:07:53.494 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-26T01:07:53.494 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_comment": "", 2024-04-26T01:07:53.494 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_createby": "", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_encoding": "", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_flag": "false", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_origin": "", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_tracker": "", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_trust_forwarded_https": "false", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_usage_max_shards": "32", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_usage_max_user_shards": "1", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_use_opa_authz": "false", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_counters_cache": "false", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-26T01:07:53.495 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-26T01:07:53.496 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_max_buckets": "1000", 2024-04-26T01:07:53.496 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_policies_max_num": "100", 2024-04-26T01:07:53.496 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-26T01:07:53.496 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-26T01:07:53.496 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-26T01:07:53.496 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-26T01:07:53.496 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_unique_email": "true", 2024-04-26T01:07:53.496 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_verify_ssl": "true", 2024-04-26T01:07:53.496 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-26T01:07:53.496 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_zone": "", 2024-04-26T01:07:53.496 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_zone_id": "", 2024-04-26T01:07:53.496 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-26T01:07:53.496 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_zonegroup": "", 2024-04-26T01:07:53.497 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_zonegroup_id": "", 2024-04-26T01:07:53.497 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-26T01:07:53.497 INFO:teuthology.orchestra.run.smithi008.stdout: "rgwlc_auto_session_clear": "true", 2024-04-26T01:07:53.497 INFO:teuthology.orchestra.run.smithi008.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-26T01:07:53.497 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_block_size": "4096", 2024-04-26T01:07:53.497 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-26T01:07:53.497 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-26T01:07:53.497 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-26T01:07:53.497 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-26T01:07:53.497 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-26T01:07:53.497 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cache_size": "536870912", 2024-04-26T01:07:53.497 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-26T01:07:53.497 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-26T01:07:53.498 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-26T01:07:53.498 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-26T01:07:53.498 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-26T01:07:53.498 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-26T01:07:53.498 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-26T01:07:53.498 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-26T01:07:53.498 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_index_type": "binary_search", 2024-04-26T01:07:53.498 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-26T01:07:53.498 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-26T01:07:53.498 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_partition_filters": "false", 2024-04-26T01:07:53.498 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_perf": "false", 2024-04-26T01:07:53.498 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-26T01:07:53.498 INFO:teuthology.orchestra.run.smithi008.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-26T01:07:53.498 INFO:teuthology.orchestra.run.smithi008.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "run_dir": "/var/run/ceph", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_block_create": "true", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_cache_lru_size": "67108864", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_default_max_object_size": "16777216", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_device_size": "53687091200", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_journal_batch_capacity": "16", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_main_device_type": "SSD", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-26T01:07:53.499 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-26T01:07:53.500 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-26T01:07:53.500 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-26T01:07:53.500 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_segment_size": "67108864", 2024-04-26T01:07:53.500 INFO:teuthology.orchestra.run.smithi008.stdout: "set_keepcaps": "false", 2024-04-26T01:07:53.500 INFO:teuthology.orchestra.run.smithi008.stdout: "setgroup": "", 2024-04-26T01:07:53.500 INFO:teuthology.orchestra.run.smithi008.stdout: "setuser": "", 2024-04-26T01:07:53.500 INFO:teuthology.orchestra.run.smithi008.stdout: "setuser_match_path": "", 2024-04-26T01:07:53.500 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-26T01:07:53.500 INFO:teuthology.orchestra.run.smithi008.stdout: "thp": "false", 2024-04-26T01:07:53.500 INFO:teuthology.orchestra.run.smithi008.stdout: "threadpool_default_timeout": "60", 2024-04-26T01:07:53.500 INFO:teuthology.orchestra.run.smithi008.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-26T01:07:53.500 INFO:teuthology.orchestra.run.smithi008.stdout: "throttler_perf_counter": "true" 2024-04-26T01:07:53.500 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:07:53.502 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-26T01:07:53.661 INFO:teuthology.orchestra.run.smithi008.stdout:mon.a: {} 2024-04-26T01:07:53.661 INFO:teuthology.orchestra.run.smithi008.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T01:07:53.679 INFO:teuthology.orchestra.run.smithi008.stdout:mon.b: {} 2024-04-26T01:07:53.680 INFO:teuthology.orchestra.run.smithi008.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T01:07:53.699 INFO:teuthology.orchestra.run.smithi008.stdout:mon.c: {} 2024-04-26T01:07:53.700 INFO:teuthology.orchestra.run.smithi008.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T01:07:53.718 DEBUG:teuthology.orchestra.run.smithi008:> type ceph-objectstore-tool 2024-04-26T01:07:53.724 INFO:teuthology.orchestra.run.smithi008.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-26T01:07:53.724 DEBUG:teuthology.orchestra.run.smithi012:> type ceph-objectstore-tool 2024-04-26T01:07:53.730 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-26T01:07:53.731 DEBUG:teuthology.orchestra.run.smithi116:> type ceph-objectstore-tool 2024-04-26T01:07:53.737 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-26T01:07:53.738 DEBUG:teuthology.orchestra.run.smithi164:> type ceph-objectstore-tool 2024-04-26T01:07:53.744 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-26T01:07:53.744 INFO:teuthology.run_tasks:Running task rados... 2024-04-26T01:07:53.762 INFO:tasks.rados:Beginning rados... 2024-04-26T01:07:53.762 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-04-26T01:07:53.787 INFO:tasks.rados:joining rados 2024-04-26T01:07:53.787 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-26T01:07:53.787 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-26T01:07:53.787 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T01:07:53.788 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-26T01:07:53.788 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-26T01:07:53.788 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:07:53.789 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-26T01:07:53.789 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:07:53.789 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-26T01:07:53.789 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-26T01:07:53.790 INFO:tasks.rados:clients are ['client.0'] 2024-04-26T01:07:53.790 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set shecprofile c=2 crush-failure-domain=osd k=4 m=3 name=shecprofile plugin=shec 2024-04-26T01:07:53.791 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:07:53.785+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:07:53.890 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:07:53.888+0000 7f6879827640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25268) UID: 0 2024-04-26T01:07:53.912 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:07:53.912 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:07:53.913 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-04-26T01:07:53.913 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:07:53.930 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:07:53.943 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:07:53.944 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:07:53.984 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:07:53.984 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:07:53.991 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:07:53.988+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:07:54.007 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:07:54.009 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:07:54.032 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:07:54.033 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:07:54.051 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:54.051 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:54.055 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:07:54.055 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:07:54.064 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:07:54.081 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:07:54.081 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:07:54.092 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:07:54.092+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:07:54.121 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:07:54.121 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:07:54.150 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:07:54.150 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:07:54.187 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:07:54.187 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:07:54.192 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:07:54.189+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:07:54.202 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:54.202 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:54.213 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:07:54.213 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:07:54.215 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:07:54.238 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:07:54.238 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:07:54.259 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-04-26T01:07:54.259 INFO:teuthology.orchestra.run.smithi008.stderr:load: shec 2024-04-26T01:07:54.265 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:07:54.266 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:07:54.287 INFO:tasks.rados:starting run 0 out of 1 2024-04-26T01:07:54.288 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-26T01:07:54.288 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure shecprofile 2024-04-26T01:07:54.289 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:07:54.289 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:07:54.293 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:07:54.292+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:07:54.318 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:07:54.318 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:07:54.325 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:07:54.321+0000 7f68e3fff640 0 -- 172.21.15.8:0/3625156758 >> [v2:172.21.15.116:6800/2158650188,v1:172.21.15.116:6801/2158650188] conn(0x7f68e421f660 msgr2=0x7f68e4221a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:07:54.355 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:07:54.356 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:07:54.361 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:54.362 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:54.376 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:07:54.394 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:07:54.392+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:07:54.396 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:07:54.396 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:07:54.494 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:07:54.493+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:07:54.508 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:54.508 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:54.521 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:07:54.595 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:07:54.592+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:07:54.677 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:54.677 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:54.691 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:07:54.696 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:07:54.693+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:07:54.797 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:07:54.792+0000 7f6879827640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25268) UID: 0 2024-04-26T01:07:54.800 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:54.801 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:54.813 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:07:54.898 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:07:54.893+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:07:54.936 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:54.936 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:54.950 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:07:54.998 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:07:54.993+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:07:55.074 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:55.074 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:55.089 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:07:55.099 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:07:55.097+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:07:55.200 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:07:55.196+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:07:55.208 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:55.208 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:55.222 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:07:55.301 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:07:55.300+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:07:55.340 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:55.341 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:55.354 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:07:55.401 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:07:55.397+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:07:55.472 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:55.472 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:55.486 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:07:55.533 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:07:55.529+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:07:55.603 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:07:55.597+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:07:55.621 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:55.621 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:55.635 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:07:55.725 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:07:55.721+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:07:55.805 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:07:55.804+0000 7f6879827640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25268) UID: 0 2024-04-26T01:07:55.808 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:55.809 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:55.823 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:07:55.906 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:07:55.904+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:07:55.993 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:55.993 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:56.007 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:07:56.007 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:07:56.004+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:07:56.107 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:07:56.105+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:07:56.133 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:56.133 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:56.147 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:07:56.208 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:07:56.201+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:07:56.272 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:56.272 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:56.276 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'unique_pool_0' created 2024-04-26T01:07:56.287 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-26T01:07:56.290 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:07:56.308 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-26T01:07:56.309 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:07:56.308+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:07:56.410 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:07:56.405+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:07:56.413 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:07:56.454 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:56.454 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:56.469 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:07:56.511 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:07:56.508+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:07:56.570 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:07:56.571 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:07:56.594 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:07:56.594 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:07:56.611 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:07:56.612+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:07:56.620 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:07:56.620 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:07:56.637 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:56.637 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:56.649 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:07:56.649 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:07:56.651 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:07:56.677 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:07:56.677 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:07:56.706 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:07:56.706 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:07:56.712 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:07:56.712+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:07:56.777 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:56.777 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:56.792 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:07:56.813 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:07:56.809+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:07:56.837 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:07:56.837 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:07:56.879 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:07:56.879 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:07:56.914 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:07:56.909+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:07:56.920 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:56.920 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:56.921 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:07:56.922 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:07:56.935 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:07:56.960 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:07:56.960 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:07:57.001 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:07:57.001 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:07:57.025 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:07:57.021+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:07:57.043 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:07:57.043 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:07:57.081 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:07:57.081 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:07:57.099 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:57.099 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:57.114 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:07:57.116 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:07:57.116+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:07:57.119 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:07:57.120 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:07:57.159 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:07:57.157+0000 7f6225a81640 0 -- 172.21.15.8:0/165527019 >> [v2:172.21.15.116:6800/2158650188,v1:172.21.15.116:6801/2158650188] conn(0x7f6220222ca0 msgr2=0x7f6220225090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:07:57.168 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:07:57.210 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:07:57.210 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:07:57.217 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:07:57.216+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:07:57.284 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-04-26T01:07:57.284 INFO:teuthology.orchestra.run.smithi008.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-26T01:07:57.285 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:07:57.285 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:07:57.285 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:07:57.285 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:07:57.285 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:07:57.304 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:07:57.318 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:07:57.316+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:07:57.419 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:07:57.416+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:07:57.427 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:07:57.427 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:07:57.427 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:07:57.427 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:07:57.427 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.427 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-26T01:07:57.427 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:46.050143+0000", 2024-04-26T01:07:57.427 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 11.374756211999999, 2024-04-26T01:07:57.427 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.36180093800000002, 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 1 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050143+0000", 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.429 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050143+0000", 2024-04-26T01:07:57.429 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.429 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.429 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.429 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.429 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050146+0000", 2024-04-26T01:07:57.429 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7829999999999999e-06 2024-04-26T01:07:57.429 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.429 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.429 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.429 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050160+0000", 2024-04-26T01:07:57.429 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4314000000000001e-05 2024-04-26T01:07:57.429 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.429 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.429 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.430 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050182+0000", 2024-04-26T01:07:57.430 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2407000000000001e-05 2024-04-26T01:07:57.430 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.430 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.430 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.430 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050200+0000", 2024-04-26T01:07:57.430 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7547e-05 2024-04-26T01:07:57.430 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.430 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.430 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.430 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050332+0000", 2024-04-26T01:07:57.430 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013237700000000001 2024-04-26T01:07:57.430 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.430 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.430 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for peered", 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050361+0000", 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8704000000000002e-05 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.407786+0000", 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.35742484899999999 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.408252+0000", 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00046585099999999998 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.408586+0000", 2024-04-26T01:07:57.431 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033444200000000001 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.409758+0000", 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001171718 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.411769+0000", 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0020110520000000002 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.411870+0000", 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000101409 2024-04-26T01:07:57.432 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.433 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.433 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.433 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.411944+0000", 2024-04-26T01:07:57.433 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.3485000000000005e-05 2024-04-26T01:07:57.433 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.433 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.433 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.433 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.433 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.433 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:90 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-26T01:07:57.433 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:46.601265+0000", 2024-04-26T01:07:57.433 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 10.823633989999999, 2024-04-26T01:07:57.433 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98632521799999995, 2024-04-26T01:07:57.433 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 90 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601265+0000", 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.434 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601265+0000", 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601267+0000", 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.37e-06 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601418+0000", 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015154400000000001 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601436+0000", 2024-04-26T01:07:57.435 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7666999999999999e-05 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601447+0000", 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1406999999999999e-05 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.602924+0000", 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0014766899999999999 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.603071+0000", 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014674600000000001 2024-04-26T01:07:57.436 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.437 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.437 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.437 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.603406+0000", 2024-04-26T01:07:57.437 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033568500000000001 2024-04-26T01:07:57.437 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.437 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.437 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.437 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.604963+0000", 2024-04-26T01:07:57.437 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0015563809999999999 2024-04-26T01:07:57.437 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.437 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.437 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.437 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587422+0000", 2024-04-26T01:07:57.437 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98245948699999996 2024-04-26T01:07:57.438 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.438 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.438 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.438 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587520+0000", 2024-04-26T01:07:57.438 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7650000000000002e-05 2024-04-26T01:07:57.438 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.438 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.438 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.438 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587590+0000", 2024-04-26T01:07:57.438 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.0591000000000002e-05 2024-04-26T01:07:57.438 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.438 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.438 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.438 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.439 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.439 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_repop_reply(client.4188.0:90 1.0 e24/23 ondisk, result = 0)", 2024-04-26T01:07:57.439 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:46.605376+0000", 2024-04-26T01:07:57.439 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 10.819523589999999, 2024-04-26T01:07:57.439 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98225920899999997, 2024-04-26T01:07:57.439 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.439 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.439 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:07:57.439 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.439 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.439 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.439 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605376+0000", 2024-04-26T01:07:57.439 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.439 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.439 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605376+0000", 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605377+0000", 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.407e-06 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605384+0000", 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.5299999999999999e-06 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.440 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.441 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587134+0000", 2024-04-26T01:07:57.441 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98174979699999998 2024-04-26T01:07:57.441 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.441 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.441 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.441 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587156+0000", 2024-04-26T01:07:57.441 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1704000000000001e-05 2024-04-26T01:07:57.441 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.441 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.441 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.441 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587391+0000", 2024-04-26T01:07:57.441 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000235052 2024-04-26T01:07:57.441 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.441 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.441 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587412+0000", 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0630999999999999e-05 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587635+0000", 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000223088 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:47.788185+0000", 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.6367140029999998, 2024-04-26T01:07:57.442 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.017295839, 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 113 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788185+0000", 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.443 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788185+0000", 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788187+0000", 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.66e-06 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788199+0000", 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2387e-05 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.444 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788220+0000", 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0367e-05 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788243+0000", 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3847000000000001e-05 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.798291+0000", 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010047416999999999 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.798443+0000", 2024-04-26T01:07:57.445 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000152597 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.798639+0000", 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000195458 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.801526+0000", 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00288763 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.805309+0000", 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.003782952 2024-04-26T01:07:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.805402+0000", 2024-04-26T01:07:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2262000000000005e-05 2024-04-26T01:07:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.805481+0000", 2024-04-26T01:07:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.9262000000000001e-05 2024-04-26T01:07:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:47.788273+0000", 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.6366260839999995, 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.018510559999999999, 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 114 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788273+0000", 2024-04-26T01:07:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788273+0000", 2024-04-26T01:07:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788274+0000", 2024-04-26T01:07:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.44e-06 2024-04-26T01:07:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788287+0000", 2024-04-26T01:07:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.206e-05 2024-04-26T01:07:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788308+0000", 2024-04-26T01:07:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1514000000000001e-05 2024-04-26T01:07:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788337+0000", 2024-04-26T01:07:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9159999999999999e-05 2024-04-26T01:07:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.800473+0000", 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012135488999999999 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.800605+0000", 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013235699999999999 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.800813+0000", 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00020807099999999999 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.805177+0000", 2024-04-26T01:07:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0043635380000000001 2024-04-26T01:07:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.806586+0000", 2024-04-26T01:07:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001409743 2024-04-26T01:07:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.806668+0000", 2024-04-26T01:07:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1155999999999993e-05 2024-04-26T01:07:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.806784+0000", 2024-04-26T01:07:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000116032 2024-04-26T01:07:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-26T01:07:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:47.788380+0000", 2024-04-26T01:07:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.6365195589999999, 2024-04-26T01:07:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.021175920000000001, 2024-04-26T01:07:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 115 2024-04-26T01:07:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788380+0000", 2024-04-26T01:07:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788380+0000", 2024-04-26T01:07:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788381+0000", 2024-04-26T01:07:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6440000000000001e-06 2024-04-26T01:07:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788577+0000", 2024-04-26T01:07:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019612399999999999 2024-04-26T01:07:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788597+0000", 2024-04-26T01:07:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9477e-05 2024-04-26T01:07:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788608+0000", 2024-04-26T01:07:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1324e-05 2024-04-26T01:07:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.801644+0000", 2024-04-26T01:07:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013035702999999999 2024-04-26T01:07:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.801882+0000", 2024-04-26T01:07:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000237832 2024-04-26T01:07:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.802075+0000", 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019343499999999999 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.805188+0000", 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0031130250000000002 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.809311+0000", 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0041231970000000003 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.458 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.458 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.809434+0000", 2024-04-26T01:07:57.458 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012248599999999999 2024-04-26T01:07:57.458 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.458 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.458 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.458 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.809555+0000", 2024-04-26T01:07:57.458 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000121673 2024-04-26T01:07:57.458 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.458 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.458 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.458 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.458 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.459 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-26T01:07:57.459 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:47.788631+0000", 2024-04-26T01:07:57.459 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.6362684899999991, 2024-04-26T01:07:57.459 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.021252778999999999, 2024-04-26T01:07:57.459 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.459 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.459 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.459 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.459 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.459 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.459 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 116 2024-04-26T01:07:57.459 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.459 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.459 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788631+0000", 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788631+0000", 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788632+0000", 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.13e-06 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.460 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.461 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788638+0000", 2024-04-26T01:07:57.461 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.477e-06 2024-04-26T01:07:57.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.461 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.461 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.461 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788650+0000", 2024-04-26T01:07:57.461 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1691e-05 2024-04-26T01:07:57.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.461 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.461 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.461 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788658+0000", 2024-04-26T01:07:57.461 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.0369999999999995e-06 2024-04-26T01:07:57.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.461 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.461 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.462 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.803281+0000", 2024-04-26T01:07:57.462 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.014623303000000001 2024-04-26T01:07:57.462 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.462 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.462 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.462 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.803417+0000", 2024-04-26T01:07:57.462 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013569299999999999 2024-04-26T01:07:57.462 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.462 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.462 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.462 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.803646+0000", 2024-04-26T01:07:57.462 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022879899999999999 2024-04-26T01:07:57.462 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.462 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.462 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.463 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.805195+0000", 2024-04-26T01:07:57.463 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0015492259999999999 2024-04-26T01:07:57.463 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.463 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.463 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.463 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.809675+0000", 2024-04-26T01:07:57.463 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0044796810000000001 2024-04-26T01:07:57.463 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.463 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.463 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.463 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.809802+0000", 2024-04-26T01:07:57.463 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000127253 2024-04-26T01:07:57.463 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.463 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.464 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.464 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.809883+0000", 2024-04-26T01:07:57.464 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1489000000000005e-05 2024-04-26T01:07:57.464 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.464 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.464 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.464 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.464 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.464 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-26T01:07:57.464 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:47.788674+0000", 2024-04-26T01:07:57.464 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.6362252850000001, 2024-04-26T01:07:57.464 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.021508167000000002, 2024-04-26T01:07:57.464 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.464 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.465 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.465 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.465 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.465 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.465 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 117 2024-04-26T01:07:57.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.465 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.465 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.465 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788674+0000", 2024-04-26T01:07:57.465 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.465 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.465 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788674+0000", 2024-04-26T01:07:57.466 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.466 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.466 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.466 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788675+0000", 2024-04-26T01:07:57.466 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.7300000000000005e-07 2024-04-26T01:07:57.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.466 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.466 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.466 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788682+0000", 2024-04-26T01:07:57.466 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8469999999999996e-06 2024-04-26T01:07:57.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.466 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.466 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.466 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788694+0000", 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2011000000000001e-05 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.788702+0000", 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.5530000000000004e-06 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.804259+0000", 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.015556719 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.804397+0000", 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000137983 2024-04-26T01:07:57.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.804623+0000", 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022651099999999999 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.807756+0000", 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0031325429999999998 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.809969+0000", 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0022127420000000002 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.469 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.469 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.810077+0000", 2024-04-26T01:07:57.469 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000108909 2024-04-26T01:07:57.469 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.469 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.469 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.469 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.810182+0000", 2024-04-26T01:07:57.469 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000104576 2024-04-26T01:07:57.469 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.469 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.469 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.469 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.469 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.469 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:145 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-26T01:07:57.469 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:47.981007+0000", 2024-04-26T01:07:57.470 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.4438917280000005, 2024-04-26T01:07:57.470 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.057997620999999999, 2024-04-26T01:07:57.470 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.470 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.470 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.470 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.470 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.470 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.470 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 145 2024-04-26T01:07:57.470 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.470 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.470 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.470 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.470 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981007+0000", 2024-04-26T01:07:57.470 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.471 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.471 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.471 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.471 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981007+0000", 2024-04-26T01:07:57.471 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.471 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.471 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.471 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.471 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981009+0000", 2024-04-26T01:07:57.471 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.204e-06 2024-04-26T01:07:57.471 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.471 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.471 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.471 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981146+0000", 2024-04-26T01:07:57.471 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013786000000000001 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981165+0000", 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8542999999999998e-05 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981177+0000", 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1787000000000001e-05 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.982190+0000", 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0010130180000000001 2024-04-26T01:07:57.472 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.473 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.473 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.473 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.982236+0000", 2024-04-26T01:07:57.473 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.6445e-05 2024-04-26T01:07:57.473 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.473 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.473 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.473 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.982311+0000", 2024-04-26T01:07:57.473 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.4657e-05 2024-04-26T01:07:57.473 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.473 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.473 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.473 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.983568+0000", 2024-04-26T01:07:57.473 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001257172 2024-04-26T01:07:57.473 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.474 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.474 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.474 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.038854+0000", 2024-04-26T01:07:57.474 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.055285918000000003 2024-04-26T01:07:57.474 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.474 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.474 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.474 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.038941+0000", 2024-04-26T01:07:57.474 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.6971999999999993e-05 2024-04-26T01:07:57.474 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.474 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.474 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.474 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.039005+0000", 2024-04-26T01:07:57.474 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.4045000000000001e-05 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:48.185361+0000", 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.2395382870000002, 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.015757027, 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 158 2024-04-26T01:07:57.475 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185361+0000", 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185361+0000", 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185362+0000", 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.263e-06 2024-04-26T01:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185552+0000", 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00018956100000000001 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185576+0000", 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4233999999999999e-05 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185594+0000", 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7663e-05 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.477 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.478 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.478 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.191479+0000", 2024-04-26T01:07:57.478 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0058854379999999998 2024-04-26T01:07:57.478 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.478 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.478 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.478 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.191601+0000", 2024-04-26T01:07:57.478 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000121963 2024-04-26T01:07:57.478 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.478 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.478 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.478 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.191804+0000", 2024-04-26T01:07:57.478 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000202781 2024-04-26T01:07:57.478 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.194732+0000", 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002928412 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.200978+0000", 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0062459059999999999 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.201056+0000", 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8144000000000003e-05 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.479 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.480 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.201118+0000", 2024-04-26T01:07:57.480 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.1661999999999993e-05 2024-04-26T01:07:57.480 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.480 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.480 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.480 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.480 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-26T01:07:57.480 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:48.185697+0000", 2024-04-26T01:07:57.480 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.2392020440000007, 2024-04-26T01:07:57.480 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.016998415999999999, 2024-04-26T01:07:57.480 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.480 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.480 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.480 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.481 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.481 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.481 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 160 2024-04-26T01:07:57.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.481 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.481 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.481 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.481 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185697+0000", 2024-04-26T01:07:57.481 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.481 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.481 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.481 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185697+0000", 2024-04-26T01:07:57.481 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.482 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.482 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185698+0000", 2024-04-26T01:07:57.482 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.297e-06 2024-04-26T01:07:57.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.482 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.482 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185708+0000", 2024-04-26T01:07:57.482 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.747e-06 2024-04-26T01:07:57.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.482 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.482 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185727+0000", 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9211e-05 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185742+0000", 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.467e-05 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.193756+0000", 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0080139249999999999 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.193882+0000", 2024-04-26T01:07:57.483 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012588000000000001 2024-04-26T01:07:57.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.484 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.484 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.194082+0000", 2024-04-26T01:07:57.484 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000200504 2024-04-26T01:07:57.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.484 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.484 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.198019+0000", 2024-04-26T01:07:57.484 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0039368880000000004 2024-04-26T01:07:57.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.484 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.484 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.202537+0000", 2024-04-26T01:07:57.484 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0045181070000000004 2024-04-26T01:07:57.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.485 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.485 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.202628+0000", 2024-04-26T01:07:57.485 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.0362e-05 2024-04-26T01:07:57.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.485 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.485 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.202696+0000", 2024-04-26T01:07:57.485 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.7824999999999995e-05 2024-04-26T01:07:57.485 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.485 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.485 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.486 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-26T01:07:57.486 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:48.185767+0000", 2024-04-26T01:07:57.486 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.2391316450000005, 2024-04-26T01:07:57.486 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.017055569999999999, 2024-04-26T01:07:57.486 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.486 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.486 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.486 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.486 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.486 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.486 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 161 2024-04-26T01:07:57.486 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.486 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.487 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.487 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185767+0000", 2024-04-26T01:07:57.487 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.487 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.487 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.487 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185767+0000", 2024-04-26T01:07:57.487 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.487 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.487 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.487 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185769+0000", 2024-04-26T01:07:57.487 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2729999999999999e-06 2024-04-26T01:07:57.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.488 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.488 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.488 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185958+0000", 2024-04-26T01:07:57.488 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00018971799999999999 2024-04-26T01:07:57.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.488 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.488 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.488 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185981+0000", 2024-04-26T01:07:57.488 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2674e-05 2024-04-26T01:07:57.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.488 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.488 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.488 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.185998+0000", 2024-04-26T01:07:57.488 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6820000000000002e-05 2024-04-26T01:07:57.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.489 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.489 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.489 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.194947+0000", 2024-04-26T01:07:57.489 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0089491620000000001 2024-04-26T01:07:57.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.489 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.489 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.489 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.195074+0000", 2024-04-26T01:07:57.489 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012719000000000001 2024-04-26T01:07:57.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.489 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.489 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.489 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.195263+0000", 2024-04-26T01:07:57.489 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00018830100000000001 2024-04-26T01:07:57.490 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.490 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.490 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.490 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.198147+0000", 2024-04-26T01:07:57.490 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0028848540000000001 2024-04-26T01:07:57.490 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.490 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.490 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.490 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.202706+0000", 2024-04-26T01:07:57.490 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0045588319999999996 2024-04-26T01:07:57.490 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.490 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.490 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.490 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.202770+0000", 2024-04-26T01:07:57.491 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.3623999999999999e-05 2024-04-26T01:07:57.491 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.491 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.491 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.491 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.202823+0000", 2024-04-26T01:07:57.491 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.3121999999999997e-05 2024-04-26T01:07:57.491 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.491 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.491 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.491 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.491 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.491 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-26T01:07:57.491 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:48.186028+0000", 2024-04-26T01:07:57.492 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.2388706490000008, 2024-04-26T01:07:57.492 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.016915698999999999, 2024-04-26T01:07:57.492 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.492 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.492 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.492 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.492 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.492 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.492 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 162 2024-04-26T01:07:57.492 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.492 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.492 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.492 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.492 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186028+0000", 2024-04-26T01:07:57.493 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.493 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.493 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.493 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.493 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186028+0000", 2024-04-26T01:07:57.493 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.493 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.493 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.493 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.493 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186030+0000", 2024-04-26T01:07:57.493 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4869999999999999e-06 2024-04-26T01:07:57.493 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.493 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.493 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.493 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186041+0000", 2024-04-26T01:07:57.494 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1202999999999999e-05 2024-04-26T01:07:57.494 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.494 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.494 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.494 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186060+0000", 2024-04-26T01:07:57.494 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8677000000000001e-05 2024-04-26T01:07:57.494 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.494 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.494 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.494 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186075+0000", 2024-04-26T01:07:57.494 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5534000000000001e-05 2024-04-26T01:07:57.494 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.494 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.494 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.495 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.196518+0000", 2024-04-26T01:07:57.495 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010443081999999999 2024-04-26T01:07:57.495 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.495 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.495 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.495 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.196754+0000", 2024-04-26T01:07:57.495 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00023529900000000001 2024-04-26T01:07:57.495 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.495 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.495 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.495 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.197133+0000", 2024-04-26T01:07:57.495 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00037971199999999999 2024-04-26T01:07:57.495 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.495 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.495 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.496 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.200833+0000", 2024-04-26T01:07:57.496 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0036998080000000002 2024-04-26T01:07:57.496 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.496 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.496 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.496 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.202838+0000", 2024-04-26T01:07:57.496 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0020042950000000001 2024-04-26T01:07:57.496 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.496 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.496 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.496 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.202897+0000", 2024-04-26T01:07:57.496 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.9231000000000003e-05 2024-04-26T01:07:57.496 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.496 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.497 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.497 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.202944+0000", 2024-04-26T01:07:57.497 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.7370999999999998e-05 2024-04-26T01:07:57.497 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.497 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.497 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.497 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.497 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.497 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-26T01:07:57.497 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:48.186102+0000", 2024-04-26T01:07:57.497 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.2387970809999995, 2024-04-26T01:07:57.497 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.016981388, 2024-04-26T01:07:57.497 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.497 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.497 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.498 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.498 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.498 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.498 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 163 2024-04-26T01:07:57.498 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.498 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.498 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.498 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.498 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186102+0000", 2024-04-26T01:07:57.498 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.498 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.498 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.498 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.498 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186102+0000", 2024-04-26T01:07:57.498 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.499 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.499 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.499 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.499 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186103+0000", 2024-04-26T01:07:57.499 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.353e-06 2024-04-26T01:07:57.499 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.499 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.499 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.499 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186115+0000", 2024-04-26T01:07:57.499 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1800999999999999e-05 2024-04-26T01:07:57.499 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.499 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.499 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.499 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186134+0000", 2024-04-26T01:07:57.500 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9029999999999999e-05 2024-04-26T01:07:57.500 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.500 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.500 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.500 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186149+0000", 2024-04-26T01:07:57.500 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.447e-05 2024-04-26T01:07:57.500 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.500 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.500 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.500 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.198232+0000", 2024-04-26T01:07:57.500 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012083353 2024-04-26T01:07:57.500 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.500 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.500 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.500 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.198360+0000", 2024-04-26T01:07:57.501 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012751599999999999 2024-04-26T01:07:57.501 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.501 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.501 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.501 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.198561+0000", 2024-04-26T01:07:57.501 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00020098100000000001 2024-04-26T01:07:57.501 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.501 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.501 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.501 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.200956+0000", 2024-04-26T01:07:57.501 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0023955869999999998 2024-04-26T01:07:57.501 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.501 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.501 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.502 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.202954+0000", 2024-04-26T01:07:57.502 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0019974279999999999 2024-04-26T01:07:57.502 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.502 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.502 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.502 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.203015+0000", 2024-04-26T01:07:57.502 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.1734000000000002e-05 2024-04-26T01:07:57.502 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.502 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.502 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.502 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.203083+0000", 2024-04-26T01:07:57.502 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.8134999999999997e-05 2024-04-26T01:07:57.502 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.502 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.502 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.503 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.503 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.503 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-26T01:07:57.503 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:48.186172+0000", 2024-04-26T01:07:57.503 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.2387274159999997, 2024-04-26T01:07:57.503 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.017622262, 2024-04-26T01:07:57.503 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.503 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.503 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.503 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.503 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.503 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.503 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 164 2024-04-26T01:07:57.503 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.504 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.504 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.504 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.504 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186172+0000", 2024-04-26T01:07:57.504 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.504 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.504 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.504 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.504 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186172+0000", 2024-04-26T01:07:57.504 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.504 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.504 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.504 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.504 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186173+0000", 2024-04-26T01:07:57.504 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3030000000000001e-06 2024-04-26T01:07:57.505 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.505 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.505 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.505 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186326+0000", 2024-04-26T01:07:57.505 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000152783 2024-04-26T01:07:57.505 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.505 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.505 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.505 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186349+0000", 2024-04-26T01:07:57.505 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2988000000000001e-05 2024-04-26T01:07:57.505 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.505 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.505 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.505 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186364+0000", 2024-04-26T01:07:57.506 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5227e-05 2024-04-26T01:07:57.506 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.506 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.506 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.506 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.199118+0000", 2024-04-26T01:07:57.506 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012753957 2024-04-26T01:07:57.506 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.506 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.506 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.506 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.199247+0000", 2024-04-26T01:07:57.506 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012872299999999999 2024-04-26T01:07:57.506 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.506 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.506 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.507 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.199539+0000", 2024-04-26T01:07:57.507 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000292286 2024-04-26T01:07:57.507 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.507 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.507 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.507 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.202517+0000", 2024-04-26T01:07:57.507 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0029782570000000002 2024-04-26T01:07:57.507 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.507 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.507 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.507 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.203652+0000", 2024-04-26T01:07:57.507 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0011343449999999999 2024-04-26T01:07:57.507 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.507 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.507 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.508 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.203732+0000", 2024-04-26T01:07:57.508 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.0215000000000004e-05 2024-04-26T01:07:57.508 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.508 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.508 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.508 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.203794+0000", 2024-04-26T01:07:57.508 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.2178e-05 2024-04-26T01:07:57.508 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.508 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.508 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.508 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.508 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.508 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-26T01:07:57.508 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:48.186435+0000", 2024-04-26T01:07:57.508 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.2384644100000006, 2024-04-26T01:07:57.509 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.017482772000000001, 2024-04-26T01:07:57.509 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.509 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.509 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.509 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.509 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.509 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.509 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 165 2024-04-26T01:07:57.509 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.509 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.509 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.509 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.509 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186435+0000", 2024-04-26T01:07:57.509 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.509 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.510 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.510 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.510 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186435+0000", 2024-04-26T01:07:57.510 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.510 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.510 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.510 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.510 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186436+0000", 2024-04-26T01:07:57.510 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4899999999999999e-06 2024-04-26T01:07:57.510 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.510 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.510 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.510 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186450+0000", 2024-04-26T01:07:57.510 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3763e-05 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186471+0000", 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0733999999999998e-05 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.186489+0000", 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8481000000000001e-05 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.201398+0000", 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.014909122 2024-04-26T01:07:57.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.512 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.512 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.512 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.201529+0000", 2024-04-26T01:07:57.512 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000130436 2024-04-26T01:07:57.512 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.512 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.512 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.512 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.201736+0000", 2024-04-26T01:07:57.512 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000207004 2024-04-26T01:07:57.512 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.512 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.512 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.512 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.203239+0000", 2024-04-26T01:07:57.512 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0015034110000000001 2024-04-26T01:07:57.512 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.513 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.513 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.513 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.203804+0000", 2024-04-26T01:07:57.513 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00056502899999999999 2024-04-26T01:07:57.513 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.513 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.513 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.513 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.203866+0000", 2024-04-26T01:07:57.513 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.2164999999999999e-05 2024-04-26T01:07:57.513 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.513 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.513 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.513 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.203918+0000", 2024-04-26T01:07:57.513 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.1137000000000002e-05 2024-04-26T01:07:57.514 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.514 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.514 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.514 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.514 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.514 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:186 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-26T01:07:57.514 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:48.354145+0000", 2024-04-26T01:07:57.514 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.0707537869999992, 2024-04-26T01:07:57.514 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.017208749999999998, 2024-04-26T01:07:57.514 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.514 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.514 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.514 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.514 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.514 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.515 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 186 2024-04-26T01:07:57.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.515 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.515 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.515 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.515 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354145+0000", 2024-04-26T01:07:57.515 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.515 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.515 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.515 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354145+0000", 2024-04-26T01:07:57.515 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.515 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.516 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.516 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354147+0000", 2024-04-26T01:07:57.516 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.761e-06 2024-04-26T01:07:57.516 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.516 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.516 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.516 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354159+0000", 2024-04-26T01:07:57.516 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1916000000000001e-05 2024-04-26T01:07:57.516 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.516 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.516 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.516 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354180+0000", 2024-04-26T01:07:57.516 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0721e-05 2024-04-26T01:07:57.516 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354200+0000", 2024-04-26T01:07:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0336999999999999e-05 2024-04-26T01:07:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.366353+0000", 2024-04-26T01:07:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012153334 2024-04-26T01:07:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.366494+0000", 2024-04-26T01:07:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014061600000000001 2024-04-26T01:07:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.518 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.518 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.518 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.366715+0000", 2024-04-26T01:07:57.518 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022136500000000001 2024-04-26T01:07:57.518 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.518 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.518 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.518 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.370128+0000", 2024-04-26T01:07:57.518 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.003412273 2024-04-26T01:07:57.518 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.518 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.518 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.518 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.371149+0000", 2024-04-26T01:07:57.518 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001020876 2024-04-26T01:07:57.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.519 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.519 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.519 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.371233+0000", 2024-04-26T01:07:57.519 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.4845000000000005e-05 2024-04-26T01:07:57.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.519 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.519 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.519 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.371354+0000", 2024-04-26T01:07:57.519 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012070600000000001 2024-04-26T01:07:57.519 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.519 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.519 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.520 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.520 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:187 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-26T01:07:57.520 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:48.354230+0000", 2024-04-26T01:07:57.520 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.0706694880000001, 2024-04-26T01:07:57.520 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.017446228000000001, 2024-04-26T01:07:57.520 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.520 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.520 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.520 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.520 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.520 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.520 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 187 2024-04-26T01:07:57.520 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.520 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.520 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.521 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.521 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354230+0000", 2024-04-26T01:07:57.521 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.521 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.521 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.521 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.521 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354230+0000", 2024-04-26T01:07:57.521 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.521 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.521 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.521 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.521 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354231+0000", 2024-04-26T01:07:57.521 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.347e-06 2024-04-26T01:07:57.521 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.521 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.522 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.522 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354243+0000", 2024-04-26T01:07:57.522 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2286999999999999e-05 2024-04-26T01:07:57.522 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.522 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.522 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.522 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354264+0000", 2024-04-26T01:07:57.522 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0216000000000001e-05 2024-04-26T01:07:57.522 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.522 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.522 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.522 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354310+0000", 2024-04-26T01:07:57.522 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.6174999999999998e-05 2024-04-26T01:07:57.522 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.523 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.523 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.523 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.367383+0000", 2024-04-26T01:07:57.523 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013073820999999999 2024-04-26T01:07:57.523 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.523 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.523 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.523 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.367510+0000", 2024-04-26T01:07:57.523 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000126086 2024-04-26T01:07:57.523 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.523 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.523 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.523 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.367733+0000", 2024-04-26T01:07:57.523 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000223215 2024-04-26T01:07:57.523 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.524 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.524 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.524 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.371378+0000", 2024-04-26T01:07:57.524 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0036454310000000002 2024-04-26T01:07:57.524 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.524 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.524 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.524 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.371447+0000", 2024-04-26T01:07:57.524 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.9189000000000005e-05 2024-04-26T01:07:57.524 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.524 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.524 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.524 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.371602+0000", 2024-04-26T01:07:57.524 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015455300000000001 2024-04-26T01:07:57.524 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.525 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.525 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.525 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.371676+0000", 2024-04-26T01:07:57.525 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.3907999999999998e-05 2024-04-26T01:07:57.525 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.525 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.525 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.525 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.525 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.525 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:188 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-26T01:07:57.525 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:48.354354+0000", 2024-04-26T01:07:57.525 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.0705447790000004, 2024-04-26T01:07:57.525 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.018172882000000001, 2024-04-26T01:07:57.525 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.525 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.526 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.526 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.526 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.526 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.526 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 188 2024-04-26T01:07:57.526 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.526 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.526 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.526 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.526 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354354+0000", 2024-04-26T01:07:57.526 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.526 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.526 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.526 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354354+0000", 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354357+0000", 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4669999999999998e-06 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354517+0000", 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000160607 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.527 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354544+0000", 2024-04-26T01:07:57.528 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6563999999999998e-05 2024-04-26T01:07:57.528 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.528 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.528 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.528 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354562+0000", 2024-04-26T01:07:57.528 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7620000000000001e-05 2024-04-26T01:07:57.528 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.528 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.528 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.528 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.368933+0000", 2024-04-26T01:07:57.528 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.01437103 2024-04-26T01:07:57.528 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.528 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.528 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.529 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.369079+0000", 2024-04-26T01:07:57.529 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014598700000000001 2024-04-26T01:07:57.529 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.529 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.529 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.529 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.369309+0000", 2024-04-26T01:07:57.529 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00023061499999999999 2024-04-26T01:07:57.529 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.529 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.529 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.529 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.371710+0000", 2024-04-26T01:07:57.529 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00240047 2024-04-26T01:07:57.529 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.529 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.529 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.530 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.372243+0000", 2024-04-26T01:07:57.530 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00053359200000000005 2024-04-26T01:07:57.530 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.530 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.530 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.530 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.372412+0000", 2024-04-26T01:07:57.530 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016840299999999999 2024-04-26T01:07:57.530 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.530 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.530 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.530 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.372527+0000", 2024-04-26T01:07:57.530 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00011552699999999999 2024-04-26T01:07:57.530 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.530 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.531 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.531 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.531 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.531 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-26T01:07:57.531 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:48.354599+0000", 2024-04-26T01:07:57.531 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.0703005359999995, 2024-04-26T01:07:57.531 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.018848752, 2024-04-26T01:07:57.531 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:07:57.531 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:07:57.531 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.531 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:07:57.531 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:07:57.531 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:07:57.531 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 189 2024-04-26T01:07:57.531 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.532 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:07:57.532 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.532 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:07:57.532 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354599+0000", 2024-04-26T01:07:57.532 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.532 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.532 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.532 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:07:57.532 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354599+0000", 2024-04-26T01:07:57.532 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:07:57.532 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.532 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.532 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:07:57.532 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354600+0000", 2024-04-26T01:07:57.532 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5859999999999999e-06 2024-04-26T01:07:57.533 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.533 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.533 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:07:57.533 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354612+0000", 2024-04-26T01:07:57.533 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2140000000000001e-05 2024-04-26T01:07:57.533 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.533 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.533 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:07:57.533 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354637+0000", 2024-04-26T01:07:57.533 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4624e-05 2024-04-26T01:07:57.533 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.533 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.533 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.533 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.354659+0000", 2024-04-26T01:07:57.533 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2116999999999998e-05 2024-04-26T01:07:57.534 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.534 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.534 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:07:57.534 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.370179+0000", 2024-04-26T01:07:57.534 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.015520413 2024-04-26T01:07:57.534 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.534 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.534 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:07:57.534 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.370311+0000", 2024-04-26T01:07:57.534 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000131466 2024-04-26T01:07:57.534 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.534 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.534 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:07:57.534 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.370511+0000", 2024-04-26T01:07:57.535 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019979399999999999 2024-04-26T01:07:57.535 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.535 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.535 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:07:57.535 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.371721+0000", 2024-04-26T01:07:57.535 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0012106040000000001 2024-04-26T01:07:57.535 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.535 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.535 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:07:57.535 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.373168+0000", 2024-04-26T01:07:57.535 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001446349 2024-04-26T01:07:57.535 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.535 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.535 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:07:57.535 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.373335+0000", 2024-04-26T01:07:57.536 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016767400000000001 2024-04-26T01:07:57.536 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:07:57.536 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:07:57.536 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:07:57.536 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.373447+0000", 2024-04-26T01:07:57.536 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00011198500000000001 2024-04-26T01:07:57.536 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.536 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.536 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.536 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:07:57.536 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:07:57.536 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:07:57.538 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:07:57.538 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:07:57.533+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:07:57.639 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:07:57.637+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:07:57.678 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:07:57.678 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:07:57.679 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:07:57.679 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:07:57.693 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:07:57.740 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:07:57.737+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:07:57.814 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:07:57.814 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:07:57.814 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:07:57.814 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:07:57.815 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:07:57.828 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:07:57.841 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:07:57.837+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:07:57.942 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:07:57.940+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:07:57.954 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:07:57.954 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:07:57.954 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:07:57.954 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:07:57.954 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.954 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:84 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'54, mlcod=24'50)", 2024-04-26T01:07:57.954 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:46.592230+0000", 2024-04-26T01:07:57.954 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.359116406, 2024-04-26T01:07:57.954 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0040816569999999998, 2024-04-26T01:07:57.954 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.592230+0000", 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.592230+0000", 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.592232+0000", 2024-04-26T01:07:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7969999999999999e-06 2024-04-26T01:07:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.592289+0000", 2024-04-26T01:07:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7210000000000003e-05 2024-04-26T01:07:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.592356+0000", 2024-04-26T01:07:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.6878999999999994e-05 2024-04-26T01:07:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.592369+0000", 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3200000000000001e-05 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.592483+0000", 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000113916 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.592518+0000", 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4993000000000002e-05 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.596232+0000", 2024-04-26T01:07:57.957 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003714293 2024-04-26T01:07:57.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.958 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.958 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:57.958 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.596312+0000", 2024-04-26T01:07:57.958 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9369000000000005e-05 2024-04-26T01:07:57.958 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.958 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:57.958 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.958 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.958 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:85 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'55, mlcod=24'50)", 2024-04-26T01:07:57.958 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:46.593880+0000", 2024-04-26T01:07:57.958 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.357465834999999, 2024-04-26T01:07:57.958 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.005014859, 2024-04-26T01:07:57.958 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.593880+0000", 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.593880+0000", 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:57.959 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.593882+0000", 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.429e-06 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.593891+0000", 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.3600000000000002e-06 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.593943+0000", 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.1845999999999998e-05 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.593955+0000", 2024-04-26T01:07:57.960 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1824000000000001e-05 2024-04-26T01:07:57.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.961 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.961 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:57.961 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.594057+0000", 2024-04-26T01:07:57.961 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000101699 2024-04-26T01:07:57.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.961 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.961 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:57.961 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.594094+0000", 2024-04-26T01:07:57.961 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7209999999999998e-05 2024-04-26T01:07:57.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.961 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.961 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:57.961 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.598827+0000", 2024-04-26T01:07:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0047336679999999999 2024-04-26T01:07:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.598895+0000", 2024-04-26T01:07:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7823000000000004e-05 2024-04-26T01:07:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:86 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'56, mlcod=24'50)", 2024-04-26T01:07:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:46.595457+0000", 2024-04-26T01:07:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.355888842000001, 2024-04-26T01:07:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0034792489999999998, 2024-04-26T01:07:57.963 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:57.963 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:57.963 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.963 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:57.963 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.963 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:57.963 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.595457+0000", 2024-04-26T01:07:57.963 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.963 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.963 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:57.963 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.595457+0000", 2024-04-26T01:07:57.963 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.964 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.964 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:57.964 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.595459+0000", 2024-04-26T01:07:57.964 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7969999999999999e-06 2024-04-26T01:07:57.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.964 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.964 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:57.964 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.595491+0000", 2024-04-26T01:07:57.964 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1585999999999999e-05 2024-04-26T01:07:57.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.964 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.964 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:57.964 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.595568+0000", 2024-04-26T01:07:57.965 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.7299999999999995e-05 2024-04-26T01:07:57.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.965 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.965 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.965 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.595584+0000", 2024-04-26T01:07:57.965 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5642999999999999e-05 2024-04-26T01:07:57.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.965 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.965 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:57.965 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.595699+0000", 2024-04-26T01:07:57.965 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000114912 2024-04-26T01:07:57.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.965 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.965 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:57.965 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.595737+0000", 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8009999999999997e-05 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.598899+0000", 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0031627539999999998 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:46.598937+0000", 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7246999999999998e-05 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.967 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:106 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'68, mlcod=25'64)", 2024-04-26T01:07:57.967 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:47.790740+0000", 2024-04-26T01:07:57.967 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.160606057000001, 2024-04-26T01:07:57.967 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0048041009999999999, 2024-04-26T01:07:57.967 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:57.967 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:57.967 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.967 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:57.967 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.967 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:57.967 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.790740+0000", 2024-04-26T01:07:57.967 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.967 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.967 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:57.968 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.790740+0000", 2024-04-26T01:07:57.968 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.968 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.968 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:57.968 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.790742+0000", 2024-04-26T01:07:57.968 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.443e-06 2024-04-26T01:07:57.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.968 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.968 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:57.968 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.790955+0000", 2024-04-26T01:07:57.968 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000213682 2024-04-26T01:07:57.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.968 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.968 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.791016+0000", 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0545999999999999e-05 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.791028+0000", 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2422999999999999e-05 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.791120+0000", 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.1396000000000004e-05 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:57.969 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.791154+0000", 2024-04-26T01:07:57.970 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.3875999999999999e-05 2024-04-26T01:07:57.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.970 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.970 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:57.970 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.795478+0000", 2024-04-26T01:07:57.970 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0043243190000000001 2024-04-26T01:07:57.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.970 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.970 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:57.970 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.795544+0000", 2024-04-26T01:07:57.970 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.6415999999999996e-05 2024-04-26T01:07:57.970 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.970 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:57.970 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.971 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.971 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:107 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'69, mlcod=25'64)", 2024-04-26T01:07:57.971 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:47.792005+0000", 2024-04-26T01:07:57.971 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.159341001, 2024-04-26T01:07:57.971 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003608652, 2024-04-26T01:07:57.971 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:57.971 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:57.971 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.971 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:57.971 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.971 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:57.971 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.792005+0000", 2024-04-26T01:07:57.971 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.972 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.972 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:57.972 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.792005+0000", 2024-04-26T01:07:57.972 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.972 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.972 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:57.972 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.792007+0000", 2024-04-26T01:07:57.972 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4640000000000001e-06 2024-04-26T01:07:57.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.972 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.972 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:57.972 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.792017+0000", 2024-04-26T01:07:57.972 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9790000000000005e-06 2024-04-26T01:07:57.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.973 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.973 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:57.973 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.792072+0000", 2024-04-26T01:07:57.973 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.5402999999999998e-05 2024-04-26T01:07:57.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.973 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.973 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.973 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.792084+0000", 2024-04-26T01:07:57.973 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1929999999999999e-05 2024-04-26T01:07:57.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.973 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.973 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:57.973 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.792180+0000", 2024-04-26T01:07:57.974 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6199000000000005e-05 2024-04-26T01:07:57.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.974 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.974 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:57.974 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.792211+0000", 2024-04-26T01:07:57.974 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0540000000000002e-05 2024-04-26T01:07:57.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.974 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.974 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:57.974 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.795548+0000", 2024-04-26T01:07:57.974 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0033375570000000001 2024-04-26T01:07:57.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.974 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.974 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.795614+0000", 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5580000000000006e-05 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:108 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'70, mlcod=25'65)", 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:47.793426+0000", 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.157920722, 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0047656520000000004, 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.793426+0000", 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.793426+0000", 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.793427+0000", 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.48e-06 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.976 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:57.977 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.793438+0000", 2024-04-26T01:07:57.977 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0953e-05 2024-04-26T01:07:57.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.977 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.977 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:57.977 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.793488+0000", 2024-04-26T01:07:57.977 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0120000000000001e-05 2024-04-26T01:07:57.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.977 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.977 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.977 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.793501+0000", 2024-04-26T01:07:57.977 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3210000000000001e-05 2024-04-26T01:07:57.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.977 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.978 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:57.978 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.793588+0000", 2024-04-26T01:07:57.978 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7099000000000001e-05 2024-04-26T01:07:57.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.978 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.978 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:57.978 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.793620+0000", 2024-04-26T01:07:57.978 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1883e-05 2024-04-26T01:07:57.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.978 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.978 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:57.978 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.798127+0000", 2024-04-26T01:07:57.978 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0045070099999999997 2024-04-26T01:07:57.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.978 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.979 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:57.979 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.798191+0000", 2024-04-26T01:07:57.979 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.3897000000000002e-05 2024-04-26T01:07:57.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.979 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:57.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.979 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.979 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:109 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'71, mlcod=25'65)", 2024-04-26T01:07:57.979 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:47.794467+0000", 2024-04-26T01:07:57.979 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.156879186999999, 2024-04-26T01:07:57.979 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0037913030000000002, 2024-04-26T01:07:57.979 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:57.979 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:57.979 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.980 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:57.980 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.980 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:57.980 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.794467+0000", 2024-04-26T01:07:57.980 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.980 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.980 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:57.980 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.794467+0000", 2024-04-26T01:07:57.980 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.980 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.980 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:57.980 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.794469+0000", 2024-04-26T01:07:57.981 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7969999999999999e-06 2024-04-26T01:07:57.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.981 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.981 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:57.981 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.794526+0000", 2024-04-26T01:07:57.981 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7185999999999998e-05 2024-04-26T01:07:57.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.981 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.981 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:57.981 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.794593+0000", 2024-04-26T01:07:57.981 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7309000000000002e-05 2024-04-26T01:07:57.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.981 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.981 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.981 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.794607+0000", 2024-04-26T01:07:57.982 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3404000000000001e-05 2024-04-26T01:07:57.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.982 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.982 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:57.982 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.794723+0000", 2024-04-26T01:07:57.982 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000115949 2024-04-26T01:07:57.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.982 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.982 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:57.982 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.794758+0000", 2024-04-26T01:07:57.982 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4925999999999997e-05 2024-04-26T01:07:57.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.982 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.982 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:57.983 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.798195+0000", 2024-04-26T01:07:57.983 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0034373120000000001 2024-04-26T01:07:57.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.983 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.983 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:57.983 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.798258+0000", 2024-04-26T01:07:57.983 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.3419999999999999e-05 2024-04-26T01:07:57.983 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.983 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:57.983 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.983 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.983 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:110 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'72, mlcod=25'65)", 2024-04-26T01:07:57.983 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:47.795655+0000", 2024-04-26T01:07:57.983 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.155690829999999, 2024-04-26T01:07:57.984 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003601421, 2024-04-26T01:07:57.984 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:57.984 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:57.984 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.984 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:57.984 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.984 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:57.984 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.795655+0000", 2024-04-26T01:07:57.984 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.984 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.984 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:57.984 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.795655+0000", 2024-04-26T01:07:57.984 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.985 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.985 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:57.985 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.795657+0000", 2024-04-26T01:07:57.985 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.33e-06 2024-04-26T01:07:57.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.985 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.985 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:57.985 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.795669+0000", 2024-04-26T01:07:57.985 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.217e-05 2024-04-26T01:07:57.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.985 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.985 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:57.985 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.795716+0000", 2024-04-26T01:07:57.985 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.6965999999999997e-05 2024-04-26T01:07:57.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.986 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.986 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.986 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.795728+0000", 2024-04-26T01:07:57.986 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2354000000000001e-05 2024-04-26T01:07:57.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.986 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.986 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:57.986 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.795814+0000", 2024-04-26T01:07:57.986 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5309000000000006e-05 2024-04-26T01:07:57.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.986 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.986 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:57.986 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.795847+0000", 2024-04-26T01:07:57.986 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.3160000000000001e-05 2024-04-26T01:07:57.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.987 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.987 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:57.987 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.799196+0000", 2024-04-26T01:07:57.987 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0033491329999999998 2024-04-26T01:07:57.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.987 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.987 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:57.987 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.799257+0000", 2024-04-26T01:07:57.987 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0999000000000003e-05 2024-04-26T01:07:57.987 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.987 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:57.987 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.988 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.988 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:113 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'75, mlcod=25'68)", 2024-04-26T01:07:57.988 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:47.799380+0000", 2024-04-26T01:07:57.988 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.151965857, 2024-04-26T01:07:57.988 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0046855619999999999, 2024-04-26T01:07:57.988 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:57.988 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:57.988 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.988 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:57.988 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.988 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:57.988 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.799380+0000", 2024-04-26T01:07:57.988 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.988 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.989 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:57.989 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.799380+0000", 2024-04-26T01:07:57.989 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.989 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:57.989 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.799382+0000", 2024-04-26T01:07:57.989 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.237e-06 2024-04-26T01:07:57.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.989 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:57.989 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.799391+0000", 2024-04-26T01:07:57.989 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.4700000000000008e-06 2024-04-26T01:07:57.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.990 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:57.990 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.799458+0000", 2024-04-26T01:07:57.990 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7286000000000006e-05 2024-04-26T01:07:57.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.990 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.990 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.990 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.799470+0000", 2024-04-26T01:07:57.990 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1409999999999999e-05 2024-04-26T01:07:57.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.990 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.990 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:57.990 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.799575+0000", 2024-04-26T01:07:57.990 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000105379 2024-04-26T01:07:57.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.991 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.991 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:57.991 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.799605+0000", 2024-04-26T01:07:57.991 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9615999999999999e-05 2024-04-26T01:07:57.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.991 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.991 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:57.991 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.803997+0000", 2024-04-26T01:07:57.991 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0043923080000000001 2024-04-26T01:07:57.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.991 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.991 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:57.991 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.804066+0000", 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.8856000000000006e-05 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:114 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'76, mlcod=25'72)", 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:47.801554+0000", 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.149791965, 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0051896959999999997, 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.992 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:57.993 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.801554+0000", 2024-04-26T01:07:57.993 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.993 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.993 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:57.993 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.801554+0000", 2024-04-26T01:07:57.993 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.993 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.993 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:57.993 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.801556+0000", 2024-04-26T01:07:57.993 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3769999999999999e-06 2024-04-26T01:07:57.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.993 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.993 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.801566+0000", 2024-04-26T01:07:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0676000000000001e-05 2024-04-26T01:07:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.801626+0000", 2024-04-26T01:07:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9147000000000001e-05 2024-04-26T01:07:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.801638+0000", 2024-04-26T01:07:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2437e-05 2024-04-26T01:07:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:57.995 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.801726+0000", 2024-04-26T01:07:57.995 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7771999999999999e-05 2024-04-26T01:07:57.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.995 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.995 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:57.995 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.801761+0000", 2024-04-26T01:07:57.995 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5549999999999997e-05 2024-04-26T01:07:57.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.995 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.995 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:57.995 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.806659+0000", 2024-04-26T01:07:57.995 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0048981939999999998 2024-04-26T01:07:57.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.995 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.995 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:57.996 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.806744+0000", 2024-04-26T01:07:57.996 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.4542999999999993e-05 2024-04-26T01:07:57.996 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.996 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:57.996 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:57.996 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.996 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.996 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:115 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'77, mlcod=25'72)", 2024-04-26T01:07:57.996 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:47.803016+0000", 2024-04-26T01:07:57.996 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.148329773, 2024-04-26T01:07:57.996 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0063906759999999996, 2024-04-26T01:07:57.996 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:57.996 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:57.996 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:57.996 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.803016+0000", 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.803016+0000", 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.803018+0000", 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.77e-06 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.997 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.998 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:57.998 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.803124+0000", 2024-04-26T01:07:57.998 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000105649 2024-04-26T01:07:57.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.998 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.998 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:57.998 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.803191+0000", 2024-04-26T01:07:57.998 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.6655999999999993e-05 2024-04-26T01:07:57.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.998 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.998 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:57.998 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.803204+0000", 2024-04-26T01:07:57.998 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3207000000000001e-05 2024-04-26T01:07:57.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.803314+0000", 2024-04-26T01:07:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000109932 2024-04-26T01:07:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.803346+0000", 2024-04-26T01:07:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1866999999999998e-05 2024-04-26T01:07:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.809319+0000", 2024-04-26T01:07:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059730820000000002 2024-04-26T01:07:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.809407+0000", 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8512999999999998e-05 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:116 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'78, mlcod=25'73)", 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:47.804377+0000", 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.146969317, 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0051151060000000003, 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:58.000 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.804377+0000", 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.804377+0000", 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.804378+0000", 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.533e-06 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.002 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:58.002 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.804388+0000", 2024-04-26T01:07:58.002 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.8870000000000004e-06 2024-04-26T01:07:58.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.002 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.002 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:58.002 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.804439+0000", 2024-04-26T01:07:58.002 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0753e-05 2024-04-26T01:07:58.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.002 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.002 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:58.002 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.804453+0000", 2024-04-26T01:07:58.002 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4269e-05 2024-04-26T01:07:58.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.003 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:58.003 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.804560+0000", 2024-04-26T01:07:58.003 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010693599999999999 2024-04-26T01:07:58.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.003 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:58.003 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.804593+0000", 2024-04-26T01:07:58.003 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.2280000000000003e-05 2024-04-26T01:07:58.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.003 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:58.003 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.809412+0000", 2024-04-26T01:07:58.003 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0048195449999999997 2024-04-26T01:07:58.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.809492+0000", 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9902999999999997e-05 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:117 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'79, mlcod=25'73)", 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:47.805411+0000", 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.145935338999999, 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0041557670000000003, 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:58.004 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:58.005 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:58.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.005 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:58.005 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.805411+0000", 2024-04-26T01:07:58.005 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.005 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:58.005 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.805411+0000", 2024-04-26T01:07:58.005 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.005 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:58.005 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.805412+0000", 2024-04-26T01:07:58.006 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5600000000000001e-06 2024-04-26T01:07:58.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.006 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.006 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:58.006 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.805424+0000", 2024-04-26T01:07:58.006 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1147e-05 2024-04-26T01:07:58.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.006 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.006 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:58.006 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.805482+0000", 2024-04-26T01:07:58.006 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7868999999999998e-05 2024-04-26T01:07:58.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.006 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.006 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:58.007 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.805493+0000", 2024-04-26T01:07:58.007 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1973999999999999e-05 2024-04-26T01:07:58.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.007 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:58.007 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.805590+0000", 2024-04-26T01:07:58.007 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6952000000000003e-05 2024-04-26T01:07:58.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.007 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:58.007 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.805623+0000", 2024-04-26T01:07:58.007 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.2273000000000001e-05 2024-04-26T01:07:58.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.007 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:58.008 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.809499+0000", 2024-04-26T01:07:58.008 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0038761529999999998 2024-04-26T01:07:58.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.008 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.008 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:58.008 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.809567+0000", 2024-04-26T01:07:58.008 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7838999999999999e-05 2024-04-26T01:07:58.008 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.008 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:58.008 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.008 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.008 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:145 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'103, mlcod=25'98)", 2024-04-26T01:07:58.008 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:47.983085+0000", 2024-04-26T01:07:58.008 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 9.9682613010000001, 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.055794923000000003, 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983085+0000", 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983085+0000", 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983086+0000", 2024-04-26T01:07:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9699999999999994e-07 2024-04-26T01:07:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983194+0000", 2024-04-26T01:07:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107569 2024-04-26T01:07:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037255+0000", 2024-04-26T01:07:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.054061302999999998 2024-04-26T01:07:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037268+0000", 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3477e-05 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037378+0000", 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000110089 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037414+0000", 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5123000000000002e-05 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.011 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.038790+0000", 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0013768820000000001 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.038880+0000", 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9482999999999997e-05 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:178 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'128, mlcod=26'125)", 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:48.355748+0000", 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 9.5955986650000007, 2024-04-26T01:07:58.012 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003622704, 2024-04-26T01:07:58.013 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:58.013 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:58.013 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:58.013 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:58.013 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.013 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:58.013 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.355748+0000", 2024-04-26T01:07:58.013 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.013 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.013 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:58.013 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.355748+0000", 2024-04-26T01:07:58.013 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.013 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.014 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:58.014 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.355749+0000", 2024-04-26T01:07:58.014 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.843e-06 2024-04-26T01:07:58.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.014 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.014 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:58.014 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.355761+0000", 2024-04-26T01:07:58.014 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1756999999999999e-05 2024-04-26T01:07:58.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.014 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.014 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:58.014 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.355824+0000", 2024-04-26T01:07:58.014 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.2621999999999998e-05 2024-04-26T01:07:58.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.355846+0000", 2024-04-26T01:07:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2532999999999999e-05 2024-04-26T01:07:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.355937+0000", 2024-04-26T01:07:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.0849999999999999e-05 2024-04-26T01:07:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.355972+0000", 2024-04-26T01:07:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4819e-05 2024-04-26T01:07:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.359258+0000", 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003285531 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.359370+0000", 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000112749 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:179 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'129, mlcod=26'125)", 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:48.356765+0000", 2024-04-26T01:07:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 9.5945811299999999, 2024-04-26T01:07:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0049207469999999996, 2024-04-26T01:07:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.356765+0000", 2024-04-26T01:07:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.356765+0000", 2024-04-26T01:07:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.356767+0000", 2024-04-26T01:07:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3829999999999999e-06 2024-04-26T01:07:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.356827+0000", 2024-04-26T01:07:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0072999999999999e-05 2024-04-26T01:07:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.356893+0000", 2024-04-26T01:07:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.6283000000000002e-05 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.356907+0000", 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3827e-05 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.357011+0000", 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000103929 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.357044+0000", 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.3296000000000001e-05 2024-04-26T01:07:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.361579+0000", 2024-04-26T01:07:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0045351039999999999 2024-04-26T01:07:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.361686+0000", 2024-04-26T01:07:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000106852 2024-04-26T01:07:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:180 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'130, mlcod=26'125)", 2024-04-26T01:07:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:48.358013+0000", 2024-04-26T01:07:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 9.5933330300000002, 2024-04-26T01:07:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0047561649999999997, 2024-04-26T01:07:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.358013+0000", 2024-04-26T01:07:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.358013+0000", 2024-04-26T01:07:58.022 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.022 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:58.022 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.358015+0000", 2024-04-26T01:07:58.022 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.77e-06 2024-04-26T01:07:58.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.022 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:58.022 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.358027+0000", 2024-04-26T01:07:58.022 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1579999999999999e-05 2024-04-26T01:07:58.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.022 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.358092+0000", 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5655999999999996e-05 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.358106+0000", 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3636e-05 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.358781+0000", 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00067480899999999996 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:58.023 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.358814+0000", 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.3249e-05 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.362691+0000", 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0038774920000000002 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.362769+0000", 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.7972999999999994e-05 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:203 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'149, mlcod=26'145)", 2024-04-26T01:07:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:50.507819+0000", 2024-04-26T01:07:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 7.4435274979999999, 2024-04-26T01:07:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0039349720000000001, 2024-04-26T01:07:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.507819+0000", 2024-04-26T01:07:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.507819+0000", 2024-04-26T01:07:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.507820+0000", 2024-04-26T01:07:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.164e-06 2024-04-26T01:07:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.508089+0000", 2024-04-26T01:07:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00026955600000000001 2024-04-26T01:07:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.508126+0000", 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6547000000000001e-05 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.508134+0000", 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9559999999999997e-06 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.508185+0000", 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0875999999999999e-05 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.508198+0000", 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3094e-05 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.511698+0000", 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0034996939999999998 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.511754+0000", 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.6085000000000003e-05 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:204 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'150, mlcod=26'145)", 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:50.509430+0000", 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 7.4419158349999996, 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.005268745, 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.509430+0000", 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.509430+0000", 2024-04-26T01:07:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.509431+0000", 2024-04-26T01:07:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.879999999999999e-07 2024-04-26T01:07:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.509442+0000", 2024-04-26T01:07:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0226e-05 2024-04-26T01:07:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.509473+0000", 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1544000000000002e-05 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.509480+0000", 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.0700000000000001e-06 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.509533+0000", 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2358000000000003e-05 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:58.031 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.509551+0000", 2024-04-26T01:07:58.032 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8655e-05 2024-04-26T01:07:58.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.032 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.032 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:58.032 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.514663+0000", 2024-04-26T01:07:58.032 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0051116679999999998 2024-04-26T01:07:58.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.032 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.032 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:58.032 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.514699+0000", 2024-04-26T01:07:58.032 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6235999999999997e-05 2024-04-26T01:07:58.032 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.032 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:58.032 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.033 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.033 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:205 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'151, mlcod=26'145)", 2024-04-26T01:07:58.033 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:50.510430+0000", 2024-04-26T01:07:58.033 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 7.4409162230000003, 2024-04-26T01:07:58.033 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.004294951, 2024-04-26T01:07:58.033 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:58.033 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:58.033 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:58.033 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:58.033 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.033 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:58.033 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.510430+0000", 2024-04-26T01:07:58.033 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.034 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.034 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:58.034 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.510430+0000", 2024-04-26T01:07:58.034 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:58.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.034 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.034 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:58.034 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.510431+0000", 2024-04-26T01:07:58.034 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9400000000000005e-07 2024-04-26T01:07:58.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.034 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.034 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:58.034 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.510436+0000", 2024-04-26T01:07:58.035 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9359999999999998e-06 2024-04-26T01:07:58.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.035 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.035 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:58.035 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.510464+0000", 2024-04-26T01:07:58.035 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8404000000000001e-05 2024-04-26T01:07:58.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.035 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.035 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:58.035 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.510470+0000", 2024-04-26T01:07:58.035 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.8660000000000002e-06 2024-04-26T01:07:58.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.036 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.036 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:58.036 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.510510+0000", 2024-04-26T01:07:58.036 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.0485999999999998e-05 2024-04-26T01:07:58.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.036 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.036 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:58.036 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.510524+0000", 2024-04-26T01:07:58.036 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3915e-05 2024-04-26T01:07:58.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.036 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.036 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:58.036 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.514701+0000", 2024-04-26T01:07:58.037 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0041771569999999999 2024-04-26T01:07:58.037 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:58.037 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:58.037 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:58.037 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:50.514725+0000", 2024-04-26T01:07:58.037 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3592999999999999e-05 2024-04-26T01:07:58.037 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.037 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:58.037 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.037 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:58.037 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:58.037 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:07:58.038 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:07:58.043 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:07:58.041+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:07:58.143 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:07:58.140+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:07:58.206 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:07:58.206 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:07:58.206 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:07:58.206 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:07:58.222 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:07:58.244 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:07:58.241+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:07:58.320 DEBUG:teuthology.orchestra.run.smithi164:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2024-04-26T01:07:58.341 INFO:tasks.rados.rados.0.smithi164.stdout:adding op weight read -> 100 2024-04-26T01:07:58.342 INFO:tasks.rados.rados.0.smithi164.stdout:adding op weight delete -> 50 2024-04-26T01:07:58.342 INFO:tasks.rados.rados.0.smithi164.stdout:adding op weight snap_create -> 50 2024-04-26T01:07:58.342 INFO:tasks.rados.rados.0.smithi164.stdout:adding op weight snap_remove -> 50 2024-04-26T01:07:58.342 INFO:tasks.rados.rados.0.smithi164.stdout:adding op weight rollback -> 50 2024-04-26T01:07:58.342 INFO:tasks.rados.rados.0.smithi164.stdout:adding op weight setattr -> 25 2024-04-26T01:07:58.342 INFO:tasks.rados.rados.0.smithi164.stdout:adding op weight rmattr -> 25 2024-04-26T01:07:58.342 INFO:tasks.rados.rados.0.smithi164.stdout:adding op weight copy_from -> 50 2024-04-26T01:07:58.342 INFO:tasks.rados.rados.0.smithi164.stdout:adding op weight append -> 100 2024-04-26T01:07:58.342 INFO:tasks.rados.rados.0.smithi164.stdout:ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-26T01:07:58.342 INFO:tasks.rados.rados.0.smithi164.stdout:Configuration: 2024-04-26T01:07:58.342 INFO:tasks.rados.rados.0.smithi164.stdout: Number of operations: 400 2024-04-26T01:07:58.342 INFO:tasks.rados.rados.0.smithi164.stdout: Number of objects: 50 2024-04-26T01:07:58.342 INFO:tasks.rados.rados.0.smithi164.stdout: Max in flight operations: 16 2024-04-26T01:07:58.343 INFO:tasks.rados.rados.0.smithi164.stdout: Object size (in bytes): 4000000 2024-04-26T01:07:58.343 INFO:tasks.rados.rados.0.smithi164.stdout: Write stride min: 400000 2024-04-26T01:07:58.343 INFO:tasks.rados.rados.0.smithi164.stdout: Write stride max: 800000 2024-04-26T01:07:58.345 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:07:58.344+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:07:58.390 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:07:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:07:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:07:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:07:58.391 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:07:58.406 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:07:58.446 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:07:58.441+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:07:58.547 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:07:58.544+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:07:58.570 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:07:58.570 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:07:58.571 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:07:58.571 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [] 2024-04-26T01:07:58.571 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:07:58.584 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:07:58.648 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:07:58.644+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:07:58.701 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:07:58.701 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:07:58.701 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:07:58.701 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:07:58.716 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:07:58.748 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:07:58.748+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:07:58.789 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-26T01:07:58.789 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T01:07:58.789 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-04-26T01:07:58.789 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 16 2024-04-26T01:07:58.789 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:07:58.849 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:07:58.848+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:07:58.885 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:07:58.885 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:07:58.885 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:07:58.885 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:07:58.885 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:07:58.900 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:07:58.950 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:07:58.944+0000 7f6879827640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25268) UID: 0 2024-04-26T01:07:59.051 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:07:59.048+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:07:59.068 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:07:59.068 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:07:59.068 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:07:59.068 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [] 2024-04-26T01:07:59.068 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:07:59.082 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:07:59.114 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:07:59.114 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-26T01:07:59.133 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-26T01:07:58.295072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":11,"kb":1509949440,"kb_used":445044,"kb_used_data":4288,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504396,"statfs":{"total":1546188226560,"available":1545732501504,"internally_reserved":0,"allocated":4390912,"data_stored":2933622,"data_compressed":270718,"data_compressed_allocated":434176,"data_compressed_original":1728512,"omap_allocated":25522,"internal_metadata":450993230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032982"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"27'157","reported_seq":214,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-26T01:07:54.925084+0000","last_change":"2024-04-26T01:07:46.409257+0000","last_active":"2024-04-26T01:07:54.925084+0000","last_peered":"2024-04-26T01:07:54.925084+0000","last_clean":"2024-04-26T01:07:54.925084+0000","last_became_active":"2024-04-26T01:07:46.407301+0000","last_became_peered":"2024-04-26T01:07:46.407301+0000","last_unstale":"2024-04-26T01:07:54.925084+0000","last_undegraded":"2024-04-26T01:07:54.925084+0000","last_fullsized":"2024-04-26T01:07:54.925084+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_clean_scrub_stamp":"2024-04-26T01:07:45.031420+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.371437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-26T01:07:54.264162+0000","last_change":"2024-04-26T01:07:48.145129+0000","last_active":"2024-04-26T01:07:54.264162+0000","last_peered":"2024-04-26T01:07:54.264162+0000","last_clean":"2024-04-26T01:07:54.264162+0000","last_became_active":"2024-04-26T01:07:48.144313+0000","last_became_peered":"2024-04-26T01:07:48.144313+0000","last_unstale":"2024-04-26T01:07:54.264162+0000","last_undegraded":"2024-04-26T01:07:54.264162+0000","last_fullsized":"2024-04-26T01:07:54.264162+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:03.023694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-26T01:07:55.031065+0000","last_change":"2024-04-26T01:07:48.149275+0000","last_active":"2024-04-26T01:07:55.031065+0000","last_peered":"2024-04-26T01:07:55.031065+0000","last_clean":"2024-04-26T01:07:55.031065+0000","last_became_active":"2024-04-26T01:07:48.148395+0000","last_became_peered":"2024-04-26T01:07:48.148395+0000","last_unstale":"2024-04-26T01:07:55.031065+0000","last_undegraded":"2024-04-26T01:07:55.031065+0000","last_fullsized":"2024-04-26T01:07:55.031065+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:49.139938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-26T01:07:54.925157+0000","last_change":"2024-04-26T01:07:48.146870+0000","last_active":"2024-04-26T01:07:54.925157+0000","last_peered":"2024-04-26T01:07:54.925157+0000","last_clean":"2024-04-26T01:07:54.925157+0000","last_became_active":"2024-04-26T01:07:48.145991+0000","last_became_peered":"2024-04-26T01:07:48.145991+0000","last_unstale":"2024-04-26T01:07:54.925157+0000","last_undegraded":"2024-04-26T01:07:54.925157+0000","last_fullsized":"2024-04-26T01:07:54.925157+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:13.523679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-26T01:07:55.031065+0000","last_change":"2024-04-26T01:07:48.147756+0000","last_active":"2024-04-26T01:07:55.031065+0000","last_peered":"2024-04-26T01:07:55.031065+0000","last_clean":"2024-04-26T01:07:55.031065+0000","last_became_active":"2024-04-26T01:07:48.147003+0000","last_became_peered":"2024-04-26T01:07:48.147003+0000","last_unstale":"2024-04-26T01:07:55.031065+0000","last_undegraded":"2024-04-26T01:07:55.031065+0000","last_fullsized":"2024-04-26T01:07:55.031065+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:05.491411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.277167+0000","last_change":"2024-04-26T01:07:48.465954+0000","last_active":"2024-04-26T01:07:49.277167+0000","last_peered":"2024-04-26T01:07:49.277167+0000","last_clean":"2024-04-26T01:07:49.277167+0000","last_became_active":"2024-04-26T01:07:48.465232+0000","last_became_peered":"2024-04-26T01:07:48.465232+0000","last_unstale":"2024-04-26T01:07:49.277167+0000","last_undegraded":"2024-04-26T01:07:49.277167+0000","last_fullsized":"2024-04-26T01:07:49.277167+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:49.270720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.900376+0000","last_change":"2024-04-26T01:07:48.145645+0000","last_active":"2024-04-26T01:07:49.900376+0000","last_peered":"2024-04-26T01:07:49.900376+0000","last_clean":"2024-04-26T01:07:49.900376+0000","last_became_active":"2024-04-26T01:07:48.145376+0000","last_became_peered":"2024-04-26T01:07:48.145376+0000","last_unstale":"2024-04-26T01:07:49.900376+0000","last_undegraded":"2024-04-26T01:07:49.900376+0000","last_fullsized":"2024-04-26T01:07:49.900376+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.310599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.277344+0000","last_change":"2024-04-26T01:07:48.145909+0000","last_active":"2024-04-26T01:07:49.277344+0000","last_peered":"2024-04-26T01:07:49.277344+0000","last_clean":"2024-04-26T01:07:49.277344+0000","last_became_active":"2024-04-26T01:07:48.145512+0000","last_became_peered":"2024-04-26T01:07:48.145512+0000","last_unstale":"2024-04-26T01:07:49.277344+0000","last_undegraded":"2024-04-26T01:07:49.277344+0000","last_fullsized":"2024-04-26T01:07:49.277344+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:16.344494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:07:56.266998+0000","last_change":"2024-04-26T01:07:56.266998+0000","last_active":"2024-04-26T01:07:56.266998+0000","last_peered":"2024-04-26T01:07:56.266998+0000","last_clean":"2024-04-26T01:07:56.266998+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:07:56.266998+0000","last_undegraded":"2024-04-26T01:07:56.266998+0000","last_fullsized":"2024-04-26T01:07:56.266998+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:07:49.687752+0000","last_change":"2024-04-26T01:07:48.149017+0000","last_active":"2024-04-26T01:07:49.687752+0000","last_peered":"2024-04-26T01:07:49.687752+0000","last_clean":"2024-04-26T01:07:49.687752+0000","last_became_active":"2024-04-26T01:07:48.148023+0000","last_became_peered":"2024-04-26T01:07:48.148023+0000","last_unstale":"2024-04-26T01:07:49.687752+0000","last_undegraded":"2024-04-26T01:07:49.687752+0000","last_fullsized":"2024-04-26T01:07:49.687752+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:07.554062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":1},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":23216,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":21,"seq":90194313222,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":323584,"data_stored":642376,"data_compressed":25965,"data_compressed_allocated":61440,"data_compressed_original":552960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":123110,"data_compressed":17302,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":323584,"data_stored":642376,"data_compressed":25965,"data_compressed_allocated":61440,"data_compressed_original":552960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,5,6,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313220,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,4,5,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,4,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":311296,"data_stored":154722,"data_compressed":26257,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,4,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313222,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,5,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":123110,"data_compressed":17302,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":113448,"data_compressed":14357,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:07:59.134 INFO:tasks.ceph.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-04-26T01:07:59.152 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:07:59.152+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:07:59.204 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:07:59.204 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:07:59.204 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:07:59.204 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:07:59.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:07:59.241 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:07:59.253 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:07:59.249+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:07:59.294 INFO:tasks.rados.rados.0.smithi164.stdout:1: write initial oid 1 2024-04-26T01:07:59.295 INFO:tasks.rados.rados.0.smithi164.stdout:1: seq_num 0 ranges {0=32768} 2024-04-26T01:07:59.295 INFO:tasks.rados.rados.0.smithi164.stdout:1: writing smithi16428812-1 from 0 to 32768 tid 1 2024-04-26T01:07:59.295 INFO:tasks.rados.rados.0.smithi164.stdout:2: write initial oid 2 2024-04-26T01:07:59.295 INFO:tasks.rados.rados.0.smithi164.stdout:2: seq_num 1 ranges {0=32768} 2024-04-26T01:07:59.296 INFO:tasks.rados.rados.0.smithi164.stdout:2: writing smithi16428812-2 from 0 to 32768 tid 1 2024-04-26T01:07:59.296 INFO:tasks.rados.rados.0.smithi164.stdout:3: write initial oid 3 2024-04-26T01:07:59.296 INFO:tasks.rados.rados.0.smithi164.stdout:3: seq_num 2 ranges {0=49152} 2024-04-26T01:07:59.296 INFO:tasks.rados.rados.0.smithi164.stdout:3: writing smithi16428812-3 from 0 to 49152 tid 1 2024-04-26T01:07:59.296 INFO:tasks.rados.rados.0.smithi164.stdout:4: write initial oid 4 2024-04-26T01:07:59.296 INFO:tasks.rados.rados.0.smithi164.stdout:4: seq_num 3 ranges {0=65536} 2024-04-26T01:07:59.297 INFO:tasks.rados.rados.0.smithi164.stdout:4: writing smithi16428812-4 from 0 to 65536 tid 1 2024-04-26T01:07:59.297 INFO:tasks.rados.rados.0.smithi164.stdout:5: write initial oid 5 2024-04-26T01:07:59.297 INFO:tasks.rados.rados.0.smithi164.stdout:5: seq_num 4 ranges {0=81920} 2024-04-26T01:07:59.297 INFO:tasks.rados.rados.0.smithi164.stdout:5: writing smithi16428812-5 from 0 to 81920 tid 1 2024-04-26T01:07:59.297 INFO:tasks.rados.rados.0.smithi164.stdout:6: write initial oid 6 2024-04-26T01:07:59.297 INFO:tasks.rados.rados.0.smithi164.stdout:6: seq_num 5 ranges {0=98304} 2024-04-26T01:07:59.298 INFO:tasks.rados.rados.0.smithi164.stdout:6: writing smithi16428812-6 from 0 to 98304 tid 1 2024-04-26T01:07:59.298 INFO:tasks.rados.rados.0.smithi164.stdout:7: write initial oid 7 2024-04-26T01:07:59.298 INFO:tasks.rados.rados.0.smithi164.stdout:7: seq_num 6 ranges {0=114688} 2024-04-26T01:07:59.299 INFO:tasks.rados.rados.0.smithi164.stdout:7: writing smithi16428812-7 from 0 to 114688 tid 1 2024-04-26T01:07:59.300 INFO:tasks.rados.rados.0.smithi164.stdout:8: write initial oid 8 2024-04-26T01:07:59.300 INFO:tasks.rados.rados.0.smithi164.stdout:8: seq_num 7 ranges {0=131072} 2024-04-26T01:07:59.301 INFO:tasks.rados.rados.0.smithi164.stdout:8: writing smithi16428812-8 from 0 to 131072 tid 1 2024-04-26T01:07:59.301 INFO:tasks.rados.rados.0.smithi164.stdout:9: write initial oid 9 2024-04-26T01:07:59.301 INFO:tasks.rados.rados.0.smithi164.stdout:9: seq_num 8 ranges {0=147456} 2024-04-26T01:07:59.304 INFO:tasks.rados.rados.0.smithi164.stdout:9: writing smithi16428812-9 from 0 to 147456 tid 1 2024-04-26T01:07:59.304 INFO:tasks.rados.rados.0.smithi164.stdout:10: write initial oid 10 2024-04-26T01:07:59.304 INFO:tasks.rados.rados.0.smithi164.stdout:10: seq_num 9 ranges {0=163840} 2024-04-26T01:07:59.308 INFO:tasks.rados.rados.0.smithi164.stdout:10: writing smithi16428812-10 from 0 to 163840 tid 1 2024-04-26T01:07:59.308 INFO:tasks.rados.rados.0.smithi164.stdout:11: write initial oid 11 2024-04-26T01:07:59.308 INFO:tasks.rados.rados.0.smithi164.stdout:11: seq_num 10 ranges {0=180224} 2024-04-26T01:07:59.312 INFO:tasks.rados.rados.0.smithi164.stdout:11: writing smithi16428812-11 from 0 to 180224 tid 1 2024-04-26T01:07:59.312 INFO:tasks.rados.rados.0.smithi164.stdout:12: write initial oid 12 2024-04-26T01:07:59.312 INFO:tasks.rados.rados.0.smithi164.stdout:12: seq_num 11 ranges {0=196608} 2024-04-26T01:07:59.316 INFO:tasks.rados.rados.0.smithi164.stdout:12: writing smithi16428812-12 from 0 to 196608 tid 1 2024-04-26T01:07:59.316 INFO:tasks.rados.rados.0.smithi164.stdout:13: write initial oid 13 2024-04-26T01:07:59.316 INFO:tasks.rados.rados.0.smithi164.stdout:13: seq_num 12 ranges {0=212992} 2024-04-26T01:07:59.321 INFO:tasks.rados.rados.0.smithi164.stdout:13: writing smithi16428812-13 from 0 to 212992 tid 1 2024-04-26T01:07:59.322 INFO:tasks.rados.rados.0.smithi164.stdout:14: write initial oid 14 2024-04-26T01:07:59.322 INFO:tasks.rados.rados.0.smithi164.stdout:14: seq_num 13 ranges {0=229376} 2024-04-26T01:07:59.323 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-26T01:07:59.326 INFO:tasks.rados.rados.0.smithi164.stdout:14: writing smithi16428812-14 from 0 to 229376 tid 1 2024-04-26T01:07:59.326 INFO:tasks.rados.rados.0.smithi164.stdout:2: finishing write tid 1 to smithi16428812-2 2024-04-26T01:07:59.326 INFO:tasks.rados.rados.0.smithi164.stdout:2: finishing write tid 2 to smithi16428812-2 2024-04-26T01:07:59.326 INFO:tasks.rados.rados.0.smithi164.stdout:1: finishing write tid 1 to smithi16428812-1 2024-04-26T01:07:59.326 INFO:tasks.rados.rados.0.smithi164.stdout:3: finishing write tid 1 to smithi16428812-3 2024-04-26T01:07:59.327 INFO:tasks.rados.rados.0.smithi164.stdout:4: finishing write tid 1 to smithi16428812-4 2024-04-26T01:07:59.327 INFO:tasks.rados.rados.0.smithi164.stdout:7: finishing write tid 1 to smithi16428812-7 2024-04-26T01:07:59.327 INFO:tasks.rados.rados.0.smithi164.stdout:1: finishing write tid 2 to smithi16428812-1 2024-04-26T01:07:59.327 INFO:tasks.rados.rados.0.smithi164.stdout:7: finishing write tid 2 to smithi16428812-7 2024-04-26T01:07:59.327 INFO:tasks.rados.rados.0.smithi164.stdout:8: finishing write tid 1 to smithi16428812-8 2024-04-26T01:07:59.327 INFO:tasks.rados.rados.0.smithi164.stdout:9: finishing write tid 1 to smithi16428812-9 2024-04-26T01:07:59.327 INFO:tasks.rados.rados.0.smithi164.stdout:8: finishing write tid 2 to smithi16428812-8 2024-04-26T01:07:59.327 INFO:tasks.rados.rados.0.smithi164.stdout:15: write initial oid 15 2024-04-26T01:07:59.327 INFO:tasks.rados.rados.0.smithi164.stdout:15: seq_num 14 ranges {0=245760} 2024-04-26T01:07:59.330 INFO:tasks.rados.rados.0.smithi164.stdout:15: writing smithi16428812-15 from 0 to 245760 tid 1 2024-04-26T01:07:59.331 INFO:tasks.rados.rados.0.smithi164.stdout:16: write initial oid 16 2024-04-26T01:07:59.331 INFO:tasks.rados.rados.0.smithi164.stdout:16: seq_num 15 ranges {0=262144} 2024-04-26T01:07:59.335 INFO:tasks.rados.rados.0.smithi164.stdout:16: writing smithi16428812-16 from 0 to 262144 tid 1 2024-04-26T01:07:59.337 INFO:tasks.rados.rados.0.smithi164.stdout:2: finishing write tid 3 to smithi16428812-2 2024-04-26T01:07:59.337 INFO:tasks.rados.rados.0.smithi164.stdout:2: oid 2 updating version 0 to 1 2024-04-26T01:07:59.337 INFO:tasks.rados.rados.0.smithi164.stdout:2: oid 2 updating version 1 to 2 2024-04-26T01:07:59.337 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-26T01:07:59.337 INFO:tasks.rados.rados.0.smithi164.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-04-26T01:07:59.337 INFO:tasks.rados.rados.0.smithi164.stdout:8: finishing write tid 3 to smithi16428812-8 2024-04-26T01:07:59.337 INFO:tasks.rados.rados.0.smithi164.stdout:8: oid 8 updating version 0 to 1 2024-04-26T01:07:59.337 INFO:tasks.rados.rados.0.smithi164.stdout:8: oid 8 updating version 1 to 2 2024-04-26T01:07:59.337 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-26T01:07:59.337 INFO:tasks.rados.rados.0.smithi164.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-04-26T01:07:59.338 INFO:tasks.rados.rados.0.smithi164.stdout:7: finishing write tid 3 to smithi16428812-7 2024-04-26T01:07:59.338 INFO:tasks.rados.rados.0.smithi164.stdout:7: oid 7 updating version 0 to 1 2024-04-26T01:07:59.338 INFO:tasks.rados.rados.0.smithi164.stdout:7: oid 7 updating version 1 to 2 2024-04-26T01:07:59.338 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 7 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-26T01:07:59.338 INFO:tasks.rados.rados.0.smithi164.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-04-26T01:07:59.338 INFO:tasks.rados.rados.0.smithi164.stdout:9: finishing write tid 2 to smithi16428812-9 2024-04-26T01:07:59.338 INFO:tasks.rados.rados.0.smithi164.stdout:5: finishing write tid 1 to smithi16428812-5 2024-04-26T01:07:59.338 INFO:tasks.rados.rados.0.smithi164.stdout:1: finishing write tid 3 to smithi16428812-1 2024-04-26T01:07:59.338 INFO:tasks.rados.rados.0.smithi164.stdout:1: oid 1 updating version 0 to 2 2024-04-26T01:07:59.338 INFO:tasks.rados.rados.0.smithi164.stdout:1: oid 1 version 2 is already newer than 1 2024-04-26T01:07:59.338 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-26T01:07:59.338 INFO:tasks.rados.rados.0.smithi164.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-04-26T01:07:59.338 INFO:tasks.rados.rados.0.smithi164.stdout:4: finishing write tid 2 to smithi16428812-4 2024-04-26T01:07:59.338 INFO:tasks.rados.rados.0.smithi164.stdout:12: finishing write tid 1 to smithi16428812-12 2024-04-26T01:07:59.339 INFO:tasks.rados.rados.0.smithi164.stdout:5: finishing write tid 2 to smithi16428812-5 2024-04-26T01:07:59.339 INFO:tasks.rados.rados.0.smithi164.stdout:10: finishing write tid 1 to smithi16428812-10 2024-04-26T01:07:59.339 INFO:tasks.rados.rados.0.smithi164.stdout:12: finishing write tid 2 to smithi16428812-12 2024-04-26T01:07:59.339 INFO:tasks.rados.rados.0.smithi164.stdout:3: finishing write tid 2 to smithi16428812-3 2024-04-26T01:07:59.339 INFO:tasks.rados.rados.0.smithi164.stdout:3: finishing write tid 3 to smithi16428812-3 2024-04-26T01:07:59.339 INFO:tasks.rados.rados.0.smithi164.stdout:3: oid 3 updating version 0 to 2 2024-04-26T01:07:59.339 INFO:tasks.rados.rados.0.smithi164.stdout:3: oid 3 version 2 is already newer than 1 2024-04-26T01:07:59.339 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-26T01:07:59.339 INFO:tasks.rados.rados.0.smithi164.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-04-26T01:07:59.339 INFO:tasks.rados.rados.0.smithi164.stdout:10: finishing write tid 2 to smithi16428812-10 2024-04-26T01:07:59.339 INFO:tasks.rados.rados.0.smithi164.stdout:1: done (15 left) 2024-04-26T01:07:59.339 INFO:tasks.rados.rados.0.smithi164.stdout:2: done (14 left) 2024-04-26T01:07:59.339 INFO:tasks.rados.rados.0.smithi164.stdout:3: done (13 left) 2024-04-26T01:07:59.339 INFO:tasks.rados.rados.0.smithi164.stdout:7: done (12 left) 2024-04-26T01:07:59.340 INFO:tasks.rados.rados.0.smithi164.stdout:8: done (11 left) 2024-04-26T01:07:59.340 INFO:tasks.rados.rados.0.smithi164.stdout:17: write initial oid 17 2024-04-26T01:07:59.340 INFO:tasks.rados.rados.0.smithi164.stdout:9: finishing write tid 3 to smithi16428812-9 2024-04-26T01:07:59.340 INFO:tasks.rados.rados.0.smithi164.stdout:9: oid 9 updating version 0 to 2 2024-04-26T01:07:59.340 INFO:tasks.rados.rados.0.smithi164.stdout:9: oid 9 version 2 is already newer than 1 2024-04-26T01:07:59.340 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 9 v 2 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-26T01:07:59.340 INFO:tasks.rados.rados.0.smithi164.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-04-26T01:07:59.340 INFO:tasks.rados.rados.0.smithi164.stdout:11: finishing write tid 1 to smithi16428812-11 2024-04-26T01:07:59.340 INFO:tasks.rados.rados.0.smithi164.stdout:13: finishing write tid 1 to smithi16428812-13 2024-04-26T01:07:59.340 INFO:tasks.rados.rados.0.smithi164.stdout:17: seq_num 16 ranges {0=278528} 2024-04-26T01:07:59.341 INFO:tasks.rados.rados.0.smithi164.stdout:17: writing smithi16428812-17 from 0 to 278528 tid 1 2024-04-26T01:07:59.342 INFO:tasks.rados.rados.0.smithi164.stdout:11: finishing write tid 2 to smithi16428812-11 2024-04-26T01:07:59.342 INFO:tasks.rados.rados.0.smithi164.stdout:6: finishing write tid 1 to smithi16428812-6 2024-04-26T01:07:59.342 INFO:tasks.rados.rados.0.smithi164.stdout:12: finishing write tid 3 to smithi16428812-12 2024-04-26T01:07:59.343 INFO:tasks.rados.rados.0.smithi164.stdout:12: oid 12 updating version 0 to 4 2024-04-26T01:07:59.343 INFO:tasks.rados.rados.0.smithi164.stdout:12: oid 12 version 4 is already newer than 3 2024-04-26T01:07:59.343 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 12 v 4 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-26T01:07:59.343 INFO:tasks.rados.rados.0.smithi164.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-04-26T01:07:59.343 INFO:tasks.rados.rados.0.smithi164.stdout:14: finishing write tid 1 to smithi16428812-14 2024-04-26T01:07:59.343 INFO:tasks.rados.rados.0.smithi164.stdout:10: finishing write tid 3 to smithi16428812-10 2024-04-26T01:07:59.343 INFO:tasks.rados.rados.0.smithi164.stdout:10: oid 10 updating version 0 to 1 2024-04-26T01:07:59.343 INFO:tasks.rados.rados.0.smithi164.stdout:10: oid 10 updating version 1 to 2 2024-04-26T01:07:59.343 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-26T01:07:59.343 INFO:tasks.rados.rados.0.smithi164.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-04-26T01:07:59.343 INFO:tasks.rados.rados.0.smithi164.stdout:5: finishing write tid 3 to smithi16428812-5 2024-04-26T01:07:59.343 INFO:tasks.rados.rados.0.smithi164.stdout:5: oid 5 updating version 0 to 1 2024-04-26T01:07:59.343 INFO:tasks.rados.rados.0.smithi164.stdout:5: oid 5 updating version 1 to 2 2024-04-26T01:07:59.343 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-26T01:07:59.343 INFO:tasks.rados.rados.0.smithi164.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-04-26T01:07:59.344 INFO:tasks.rados.rados.0.smithi164.stdout:5: done (11 left) 2024-04-26T01:07:59.344 INFO:tasks.rados.rados.0.smithi164.stdout:9: done (10 left) 2024-04-26T01:07:59.344 INFO:tasks.rados.rados.0.smithi164.stdout:10: done (9 left) 2024-04-26T01:07:59.344 INFO:tasks.rados.rados.0.smithi164.stdout:12: done (8 left) 2024-04-26T01:07:59.344 INFO:tasks.rados.rados.0.smithi164.stdout:18: write initial oid 18 2024-04-26T01:07:59.344 INFO:tasks.rados.rados.0.smithi164.stdout:18: seq_num 17 ranges {0=294912} 2024-04-26T01:07:59.346 INFO:tasks.rados.rados.0.smithi164.stdout:18: writing smithi16428812-18 from 0 to 294912 tid 1 2024-04-26T01:07:59.346 INFO:tasks.rados.rados.0.smithi164.stdout:19: write initial oid 19 2024-04-26T01:07:59.347 INFO:tasks.rados.rados.0.smithi164.stdout:19: seq_num 18 ranges {0=311296} 2024-04-26T01:07:59.351 INFO:tasks.rados.rados.0.smithi164.stdout:19: writing smithi16428812-19 from 0 to 311296 tid 1 2024-04-26T01:07:59.351 INFO:tasks.rados.rados.0.smithi164.stdout:20: write initial oid 20 2024-04-26T01:07:59.351 INFO:tasks.rados.rados.0.smithi164.stdout:20: seq_num 19 ranges {0=327680} 2024-04-26T01:07:59.353 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:07:59.349+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:07:59.355 INFO:tasks.rados.rados.0.smithi164.stdout:20: writing smithi16428812-20 from 0 to 327680 tid 1 2024-04-26T01:07:59.355 INFO:tasks.rados.rados.0.smithi164.stdout:21: write initial oid 21 2024-04-26T01:07:59.356 INFO:tasks.rados.rados.0.smithi164.stdout:21: seq_num 20 ranges {0=344064} 2024-04-26T01:07:59.360 INFO:tasks.rados.rados.0.smithi164.stdout:21: writing smithi16428812-21 from 0 to 344064 tid 1 2024-04-26T01:07:59.360 INFO:tasks.rados.rados.0.smithi164.stdout:22: write initial oid 22 2024-04-26T01:07:59.360 INFO:tasks.rados.rados.0.smithi164.stdout:22: seq_num 21 ranges {0=360448} 2024-04-26T01:07:59.364 INFO:tasks.rados.rados.0.smithi164.stdout:22: writing smithi16428812-22 from 0 to 360448 tid 1 2024-04-26T01:07:59.364 INFO:tasks.rados.rados.0.smithi164.stdout:23: write initial oid 23 2024-04-26T01:07:59.364 INFO:tasks.rados.rados.0.smithi164.stdout:23: seq_num 22 ranges {0=376832} 2024-04-26T01:07:59.367 INFO:tasks.rados.rados.0.smithi164.stdout:23: writing smithi16428812-23 from 0 to 376832 tid 1 2024-04-26T01:07:59.367 INFO:tasks.rados.rados.0.smithi164.stdout:24: write initial oid 24 2024-04-26T01:07:59.367 INFO:tasks.rados.rados.0.smithi164.stdout:24: seq_num 23 ranges {0=393216} 2024-04-26T01:07:59.372 INFO:tasks.rados.rados.0.smithi164.stdout:24: writing smithi16428812-24 from 0 to 393216 tid 1 2024-04-26T01:07:59.372 INFO:tasks.rados.rados.0.smithi164.stdout:25: write initial oid 25 2024-04-26T01:07:59.372 INFO:tasks.rados.rados.0.smithi164.stdout:25: seq_num 24 ranges {0=409600} 2024-04-26T01:07:59.376 INFO:tasks.rados.rados.0.smithi164.stdout:25: writing smithi16428812-25 from 0 to 409600 tid 1 2024-04-26T01:07:59.376 INFO:tasks.rados.rados.0.smithi164.stdout: waiting on 16 2024-04-26T01:07:59.376 INFO:tasks.rados.rados.0.smithi164.stdout:4: finishing write tid 3 to smithi16428812-4 2024-04-26T01:07:59.376 INFO:tasks.rados.rados.0.smithi164.stdout:4: oid 4 updating version 0 to 1 2024-04-26T01:07:59.376 INFO:tasks.rados.rados.0.smithi164.stdout:4: oid 4 updating version 1 to 2 2024-04-26T01:07:59.376 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-26T01:07:59.376 INFO:tasks.rados.rados.0.smithi164.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-04-26T01:07:59.376 INFO:tasks.rados.rados.0.smithi164.stdout:4: done (15 left) 2024-04-26T01:07:59.376 INFO:tasks.rados.rados.0.smithi164.stdout:26: write initial oid 26 2024-04-26T01:07:59.376 INFO:tasks.rados.rados.0.smithi164.stdout:26: seq_num 25 ranges {0=425984} 2024-04-26T01:07:59.380 INFO:tasks.rados.rados.0.smithi164.stdout:26: writing smithi16428812-26 from 0 to 425984 tid 1 2024-04-26T01:07:59.381 INFO:tasks.rados.rados.0.smithi164.stdout: waiting on 16 2024-04-26T01:07:59.381 INFO:tasks.rados.rados.0.smithi164.stdout:14: finishing write tid 2 to smithi16428812-14 2024-04-26T01:07:59.381 INFO:tasks.rados.rados.0.smithi164.stdout:16: finishing write tid 1 to smithi16428812-16 2024-04-26T01:07:59.381 INFO:tasks.rados.rados.0.smithi164.stdout:6: finishing write tid 2 to smithi16428812-6 2024-04-26T01:07:59.381 INFO:tasks.rados.rados.0.smithi164.stdout:16: finishing write tid 2 to smithi16428812-16 2024-04-26T01:07:59.381 INFO:tasks.rados.rados.0.smithi164.stdout:6: finishing write tid 3 to smithi16428812-6 2024-04-26T01:07:59.381 INFO:tasks.rados.rados.0.smithi164.stdout:6: oid 6 updating version 0 to 1 2024-04-26T01:07:59.381 INFO:tasks.rados.rados.0.smithi164.stdout:6: oid 6 updating version 1 to 2 2024-04-26T01:07:59.381 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-26T01:07:59.381 INFO:tasks.rados.rados.0.smithi164.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-04-26T01:07:59.381 INFO:tasks.rados.rados.0.smithi164.stdout:14: finishing write tid 3 to smithi16428812-14 2024-04-26T01:07:59.381 INFO:tasks.rados.rados.0.smithi164.stdout:14: oid 14 updating version 0 to 4 2024-04-26T01:07:59.381 INFO:tasks.rados.rados.0.smithi164.stdout:14: oid 14 version 4 is already newer than 3 2024-04-26T01:07:59.381 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 14 v 4 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-26T01:07:59.382 INFO:tasks.rados.rados.0.smithi164.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-04-26T01:07:59.382 INFO:tasks.rados.rados.0.smithi164.stdout:17: finishing write tid 1 to smithi16428812-17 2024-04-26T01:07:59.382 INFO:tasks.rados.rados.0.smithi164.stdout:17: finishing write tid 2 to smithi16428812-17 2024-04-26T01:07:59.382 INFO:tasks.rados.rados.0.smithi164.stdout:13: finishing write tid 2 to smithi16428812-13 2024-04-26T01:07:59.382 INFO:tasks.rados.rados.0.smithi164.stdout:15: finishing write tid 1 to smithi16428812-15 2024-04-26T01:07:59.382 INFO:tasks.rados.rados.0.smithi164.stdout:15: finishing write tid 2 to smithi16428812-15 2024-04-26T01:07:59.382 INFO:tasks.rados.rados.0.smithi164.stdout:18: finishing write tid 1 to smithi16428812-18 2024-04-26T01:07:59.382 INFO:tasks.rados.rados.0.smithi164.stdout:19: finishing write tid 1 to smithi16428812-19 2024-04-26T01:07:59.382 INFO:tasks.rados.rados.0.smithi164.stdout:17: finishing write tid 3 to smithi16428812-17 2024-04-26T01:07:59.382 INFO:tasks.rados.rados.0.smithi164.stdout:17: oid 17 updating version 0 to 4 2024-04-26T01:07:59.382 INFO:tasks.rados.rados.0.smithi164.stdout:17: oid 17 version 4 is already newer than 3 2024-04-26T01:07:59.382 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 17 v 4 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-26T01:07:59.382 INFO:tasks.rados.rados.0.smithi164.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-04-26T01:07:59.382 INFO:tasks.rados.rados.0.smithi164.stdout:19: finishing write tid 2 to smithi16428812-19 2024-04-26T01:07:59.383 INFO:tasks.rados.rados.0.smithi164.stdout:13: finishing write tid 3 to smithi16428812-13 2024-04-26T01:07:59.383 INFO:tasks.rados.rados.0.smithi164.stdout:13: oid 13 updating version 0 to 4 2024-04-26T01:07:59.383 INFO:tasks.rados.rados.0.smithi164.stdout:13: oid 13 version 4 is already newer than 3 2024-04-26T01:07:59.383 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 13 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-26T01:07:59.383 INFO:tasks.rados.rados.0.smithi164.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-04-26T01:07:59.383 INFO:tasks.rados.rados.0.smithi164.stdout:15: finishing write tid 3 to smithi16428812-15 2024-04-26T01:07:59.383 INFO:tasks.rados.rados.0.smithi164.stdout:15: oid 15 updating version 0 to 1 2024-04-26T01:07:59.383 INFO:tasks.rados.rados.0.smithi164.stdout:15: oid 15 updating version 1 to 2 2024-04-26T01:07:59.383 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 15 v 2 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-26T01:07:59.383 INFO:tasks.rados.rados.0.smithi164.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-04-26T01:07:59.383 INFO:tasks.rados.rados.0.smithi164.stdout:21: finishing write tid 1 to smithi16428812-21 2024-04-26T01:07:59.383 INFO:tasks.rados.rados.0.smithi164.stdout:16: finishing write tid 3 to smithi16428812-16 2024-04-26T01:07:59.383 INFO:tasks.rados.rados.0.smithi164.stdout:16: oid 16 updating version 0 to 5 2024-04-26T01:07:59.383 INFO:tasks.rados.rados.0.smithi164.stdout:16: oid 16 updating version 5 to 6 2024-04-26T01:07:59.384 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-26T01:07:59.384 INFO:tasks.rados.rados.0.smithi164.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-04-26T01:07:59.384 INFO:tasks.rados.rados.0.smithi164.stdout:19: finishing write tid 3 to smithi16428812-19 2024-04-26T01:07:59.384 INFO:tasks.rados.rados.0.smithi164.stdout:19: oid 19 updating version 0 to 6 2024-04-26T01:07:59.384 INFO:tasks.rados.rados.0.smithi164.stdout:19: oid 19 version 6 is already newer than 5 2024-04-26T01:07:59.384 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-26T01:07:59.384 INFO:tasks.rados.rados.0.smithi164.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-04-26T01:07:59.384 INFO:tasks.rados.rados.0.smithi164.stdout:22: finishing write tid 1 to smithi16428812-22 2024-04-26T01:07:59.384 INFO:tasks.rados.rados.0.smithi164.stdout:18: finishing write tid 2 to smithi16428812-18 2024-04-26T01:07:59.384 INFO:tasks.rados.rados.0.smithi164.stdout:23: finishing write tid 1 to smithi16428812-23 2024-04-26T01:07:59.384 INFO:tasks.rados.rados.0.smithi164.stdout:23: finishing write tid 2 to smithi16428812-23 2024-04-26T01:07:59.384 INFO:tasks.rados.rados.0.smithi164.stdout:18: finishing write tid 3 to smithi16428812-18 2024-04-26T01:07:59.384 INFO:tasks.rados.rados.0.smithi164.stdout:18: oid 18 updating version 0 to 4 2024-04-26T01:07:59.384 INFO:tasks.rados.rados.0.smithi164.stdout:18: oid 18 version 4 is already newer than 3 2024-04-26T01:07:59.384 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 18 v 4 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-26T01:07:59.385 INFO:tasks.rados.rados.0.smithi164.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-04-26T01:07:59.385 INFO:tasks.rados.rados.0.smithi164.stdout:6: done (15 left) 2024-04-26T01:07:59.385 INFO:tasks.rados.rados.0.smithi164.stdout:13: done (14 left) 2024-04-26T01:07:59.385 INFO:tasks.rados.rados.0.smithi164.stdout:14: done (13 left) 2024-04-26T01:07:59.385 INFO:tasks.rados.rados.0.smithi164.stdout:15: done (12 left) 2024-04-26T01:07:59.385 INFO:tasks.rados.rados.0.smithi164.stdout:16: done (11 left) 2024-04-26T01:07:59.385 INFO:tasks.rados.rados.0.smithi164.stdout:17: done (10 left) 2024-04-26T01:07:59.385 INFO:tasks.rados.rados.0.smithi164.stdout:18: done (9 left) 2024-04-26T01:07:59.385 INFO:tasks.rados.rados.0.smithi164.stdout:19: done (8 left) 2024-04-26T01:07:59.385 INFO:tasks.rados.rados.0.smithi164.stdout:27: write initial oid 27 2024-04-26T01:07:59.385 INFO:tasks.rados.rados.0.smithi164.stdout:27: seq_num 26 ranges {0=442368} 2024-04-26T01:07:59.386 INFO:tasks.rados.rados.0.smithi164.stdout:27: writing smithi16428812-27 from 0 to 442368 tid 1 2024-04-26T01:07:59.386 INFO:tasks.rados.rados.0.smithi164.stdout:28: write initial oid 28 2024-04-26T01:07:59.386 INFO:tasks.rados.rados.0.smithi164.stdout:28: seq_num 27 ranges {0=458752} 2024-04-26T01:07:59.387 INFO:tasks.rados.rados.0.smithi164.stdout:28: writing smithi16428812-28 from 0 to 458752 tid 1 2024-04-26T01:07:59.387 INFO:tasks.rados.rados.0.smithi164.stdout:29: write initial oid 29 2024-04-26T01:07:59.387 INFO:tasks.rados.rados.0.smithi164.stdout:29: seq_num 28 ranges {0=475136} 2024-04-26T01:07:59.391 INFO:tasks.rados.rados.0.smithi164.stdout:29: writing smithi16428812-29 from 0 to 475136 tid 1 2024-04-26T01:07:59.391 INFO:tasks.rados.rados.0.smithi164.stdout:30: write initial oid 30 2024-04-26T01:07:59.391 INFO:tasks.rados.rados.0.smithi164.stdout:30: seq_num 29 ranges {0=491520} 2024-04-26T01:07:59.395 INFO:tasks.rados.rados.0.smithi164.stdout:30: writing smithi16428812-30 from 0 to 491520 tid 1 2024-04-26T01:07:59.396 INFO:tasks.rados.rados.0.smithi164.stdout:31: write initial oid 31 2024-04-26T01:07:59.396 INFO:tasks.rados.rados.0.smithi164.stdout:31: seq_num 30 ranges {0=507904} 2024-04-26T01:07:59.400 INFO:tasks.rados.rados.0.smithi164.stdout:31: writing smithi16428812-31 from 0 to 507904 tid 1 2024-04-26T01:07:59.401 INFO:tasks.rados.rados.0.smithi164.stdout:32: write initial oid 32 2024-04-26T01:07:59.401 INFO:tasks.rados.rados.0.smithi164.stdout:32: seq_num 31 ranges {0=524288} 2024-04-26T01:07:59.404 INFO:tasks.rados.rados.0.smithi164.stdout:32: writing smithi16428812-32 from 0 to 524288 tid 1 2024-04-26T01:07:59.405 INFO:tasks.rados.rados.0.smithi164.stdout:33: write initial oid 33 2024-04-26T01:07:59.405 INFO:tasks.rados.rados.0.smithi164.stdout:33: seq_num 32 ranges {0=540672} 2024-04-26T01:07:59.407 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:07:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:07:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:07:59.408 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:07:59.408 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:07:59.408 INFO:tasks.rados.rados.0.smithi164.stdout:33: writing smithi16428812-33 from 0 to 540672 tid 1 2024-04-26T01:07:59.408 INFO:tasks.rados.rados.0.smithi164.stdout:34: write initial oid 34 2024-04-26T01:07:59.408 INFO:tasks.rados.rados.0.smithi164.stdout:34: seq_num 33 ranges {0=557056} 2024-04-26T01:07:59.414 INFO:tasks.rados.rados.0.smithi164.stdout:34: writing smithi16428812-34 from 0 to 557056 tid 1 2024-04-26T01:07:59.414 INFO:tasks.rados.rados.0.smithi164.stdout: waiting on 16 2024-04-26T01:07:59.414 INFO:tasks.rados.rados.0.smithi164.stdout:24: finishing write tid 1 to smithi16428812-24 2024-04-26T01:07:59.414 INFO:tasks.rados.rados.0.smithi164.stdout:24: finishing write tid 2 to smithi16428812-24 2024-04-26T01:07:59.414 INFO:tasks.rados.rados.0.smithi164.stdout:25: finishing write tid 1 to smithi16428812-25 2024-04-26T01:07:59.414 INFO:tasks.rados.rados.0.smithi164.stdout:25: finishing write tid 2 to smithi16428812-25 2024-04-26T01:07:59.414 INFO:tasks.rados.rados.0.smithi164.stdout:26: finishing write tid 1 to smithi16428812-26 2024-04-26T01:07:59.415 INFO:tasks.rados.rados.0.smithi164.stdout:26: finishing write tid 2 to smithi16428812-26 2024-04-26T01:07:59.415 INFO:tasks.rados.rados.0.smithi164.stdout:25: finishing write tid 3 to smithi16428812-25 2024-04-26T01:07:59.415 INFO:tasks.rados.rados.0.smithi164.stdout:25: oid 25 updating version 0 to 4 2024-04-26T01:07:59.415 INFO:tasks.rados.rados.0.smithi164.stdout:25: oid 25 version 4 is already newer than 3 2024-04-26T01:07:59.415 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 25 v 4 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-26T01:07:59.415 INFO:tasks.rados.rados.0.smithi164.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-04-26T01:07:59.415 INFO:tasks.rados.rados.0.smithi164.stdout:24: finishing write tid 3 to smithi16428812-24 2024-04-26T01:07:59.415 INFO:tasks.rados.rados.0.smithi164.stdout:24: oid 24 updating version 0 to 3 2024-04-26T01:07:59.415 INFO:tasks.rados.rados.0.smithi164.stdout:24: oid 24 updating version 3 to 4 2024-04-26T01:07:59.415 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 24 v 4 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-26T01:07:59.415 INFO:tasks.rados.rados.0.smithi164.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-04-26T01:07:59.415 INFO:tasks.rados.rados.0.smithi164.stdout:11: finishing write tid 3 to smithi16428812-11 2024-04-26T01:07:59.415 INFO:tasks.rados.rados.0.smithi164.stdout:11: oid 11 updating version 0 to 3 2024-04-26T01:07:59.416 INFO:tasks.rados.rados.0.smithi164.stdout:11: oid 11 updating version 3 to 4 2024-04-26T01:07:59.416 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 11 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-26T01:07:59.416 INFO:tasks.rados.rados.0.smithi164.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-04-26T01:07:59.416 INFO:tasks.rados.rados.0.smithi164.stdout:28: finishing write tid 1 to smithi16428812-28 2024-04-26T01:07:59.416 INFO:tasks.rados.rados.0.smithi164.stdout:28: finishing write tid 2 to smithi16428812-28 2024-04-26T01:07:59.416 INFO:tasks.rados.rados.0.smithi164.stdout:23: finishing write tid 3 to smithi16428812-23 2024-04-26T01:07:59.416 INFO:tasks.rados.rados.0.smithi164.stdout:23: oid 23 updating version 0 to 5 2024-04-26T01:07:59.416 INFO:tasks.rados.rados.0.smithi164.stdout:23: oid 23 updating version 5 to 6 2024-04-26T01:07:59.416 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-26T01:07:59.416 INFO:tasks.rados.rados.0.smithi164.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-04-26T01:07:59.416 INFO:tasks.rados.rados.0.smithi164.stdout:28: finishing write tid 3 to smithi16428812-28 2024-04-26T01:07:59.416 INFO:tasks.rados.rados.0.smithi164.stdout:28: oid 28 updating version 0 to 3 2024-04-26T01:07:59.416 INFO:tasks.rados.rados.0.smithi164.stdout:28: oid 28 updating version 3 to 4 2024-04-26T01:07:59.416 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 28 v 4 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-26T01:07:59.416 INFO:tasks.rados.rados.0.smithi164.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-04-26T01:07:59.417 INFO:tasks.rados.rados.0.smithi164.stdout:29: finishing write tid 1 to smithi16428812-29 2024-04-26T01:07:59.417 INFO:tasks.rados.rados.0.smithi164.stdout:29: finishing write tid 2 to smithi16428812-29 2024-04-26T01:07:59.417 INFO:tasks.rados.rados.0.smithi164.stdout:26: finishing write tid 3 to smithi16428812-26 2024-04-26T01:07:59.417 INFO:tasks.rados.rados.0.smithi164.stdout:26: oid 26 updating version 0 to 5 2024-04-26T01:07:59.417 INFO:tasks.rados.rados.0.smithi164.stdout:26: oid 26 updating version 5 to 6 2024-04-26T01:07:59.417 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-26T01:07:59.417 INFO:tasks.rados.rados.0.smithi164.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-04-26T01:07:59.417 INFO:tasks.rados.rados.0.smithi164.stdout:29: finishing write tid 3 to smithi16428812-29 2024-04-26T01:07:59.417 INFO:tasks.rados.rados.0.smithi164.stdout:29: oid 29 updating version 0 to 3 2024-04-26T01:07:59.417 INFO:tasks.rados.rados.0.smithi164.stdout:29: oid 29 updating version 3 to 4 2024-04-26T01:07:59.417 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 29 v 4 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-26T01:07:59.417 INFO:tasks.rados.rados.0.smithi164.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-04-26T01:07:59.417 INFO:tasks.rados.rados.0.smithi164.stdout:30: finishing write tid 1 to smithi16428812-30 2024-04-26T01:07:59.417 INFO:tasks.rados.rados.0.smithi164.stdout:11: done (15 left) 2024-04-26T01:07:59.418 INFO:tasks.rados.rados.0.smithi164.stdout:23: done (14 left) 2024-04-26T01:07:59.418 INFO:tasks.rados.rados.0.smithi164.stdout:24: done (13 left) 2024-04-26T01:07:59.418 INFO:tasks.rados.rados.0.smithi164.stdout:25: done (12 left) 2024-04-26T01:07:59.418 INFO:tasks.rados.rados.0.smithi164.stdout:26: done (11 left) 2024-04-26T01:07:59.418 INFO:tasks.rados.rados.0.smithi164.stdout:28: done (10 left) 2024-04-26T01:07:59.418 INFO:tasks.rados.rados.0.smithi164.stdout:29: done (9 left) 2024-04-26T01:07:59.418 INFO:tasks.rados.rados.0.smithi164.stdout:35: write initial oid 35 2024-04-26T01:07:59.418 INFO:tasks.rados.rados.0.smithi164.stdout:35: seq_num 34 ranges {0=573440} 2024-04-26T01:07:59.419 INFO:tasks.rados.rados.0.smithi164.stdout:35: writing smithi16428812-35 from 0 to 573440 tid 1 2024-04-26T01:07:59.419 INFO:tasks.rados.rados.0.smithi164.stdout:36: write initial oid 36 2024-04-26T01:07:59.419 INFO:tasks.rados.rados.0.smithi164.stdout:36: seq_num 35 ranges {0=589824} 2024-04-26T01:07:59.422 INFO:tasks.rados.rados.0.smithi164.stdout:36: writing smithi16428812-36 from 0 to 589824 tid 1 2024-04-26T01:07:59.422 INFO:tasks.rados.rados.0.smithi164.stdout:37: write initial oid 37 2024-04-26T01:07:59.422 INFO:tasks.rados.rados.0.smithi164.stdout:37: seq_num 36 ranges {0=606208} 2024-04-26T01:07:59.425 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:07:59.427 INFO:tasks.rados.rados.0.smithi164.stdout:37: writing smithi16428812-37 from 0 to 606208 tid 1 2024-04-26T01:07:59.427 INFO:tasks.rados.rados.0.smithi164.stdout:38: write initial oid 38 2024-04-26T01:07:59.427 INFO:tasks.rados.rados.0.smithi164.stdout:38: seq_num 37 ranges {0=622592} 2024-04-26T01:07:59.431 INFO:tasks.rados.rados.0.smithi164.stdout:38: writing smithi16428812-38 from 0 to 622592 tid 1 2024-04-26T01:07:59.431 INFO:tasks.rados.rados.0.smithi164.stdout:39: write initial oid 39 2024-04-26T01:07:59.431 INFO:tasks.rados.rados.0.smithi164.stdout:39: seq_num 38 ranges {0=638976} 2024-04-26T01:07:59.436 INFO:tasks.rados.rados.0.smithi164.stdout:39: writing smithi16428812-39 from 0 to 638976 tid 1 2024-04-26T01:07:59.436 INFO:tasks.rados.rados.0.smithi164.stdout:40: write initial oid 40 2024-04-26T01:07:59.436 INFO:tasks.rados.rados.0.smithi164.stdout:40: seq_num 39 ranges {0=671744} 2024-04-26T01:07:59.441 INFO:tasks.rados.rados.0.smithi164.stdout:40: writing smithi16428812-40 from 0 to 671744 tid 1 2024-04-26T01:07:59.441 INFO:tasks.rados.rados.0.smithi164.stdout:41: write initial oid 41 2024-04-26T01:07:59.441 INFO:tasks.rados.rados.0.smithi164.stdout:41: seq_num 40 ranges {0=688128} 2024-04-26T01:07:59.445 INFO:tasks.rados.rados.0.smithi164.stdout:41: writing smithi16428812-41 from 0 to 688128 tid 1 2024-04-26T01:07:59.445 INFO:tasks.rados.rados.0.smithi164.stdout: waiting on 16 2024-04-26T01:07:59.445 INFO:tasks.rados.rados.0.smithi164.stdout:31: finishing write tid 1 to smithi16428812-31 2024-04-26T01:07:59.445 INFO:tasks.rados.rados.0.smithi164.stdout:32: finishing write tid 1 to smithi16428812-32 2024-04-26T01:07:59.445 INFO:tasks.rados.rados.0.smithi164.stdout:32: finishing write tid 2 to smithi16428812-32 2024-04-26T01:07:59.446 INFO:tasks.rados.rados.0.smithi164.stdout:32: finishing write tid 3 to smithi16428812-32 2024-04-26T01:07:59.446 INFO:tasks.rados.rados.0.smithi164.stdout:32: oid 32 updating version 0 to 5 2024-04-26T01:07:59.446 INFO:tasks.rados.rados.0.smithi164.stdout:32: oid 32 updating version 5 to 6 2024-04-26T01:07:59.446 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-26T01:07:59.446 INFO:tasks.rados.rados.0.smithi164.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-04-26T01:07:59.446 INFO:tasks.rados.rados.0.smithi164.stdout:30: finishing write tid 2 to smithi16428812-30 2024-04-26T01:07:59.446 INFO:tasks.rados.rados.0.smithi164.stdout:33: finishing write tid 1 to smithi16428812-33 2024-04-26T01:07:59.446 INFO:tasks.rados.rados.0.smithi164.stdout:33: finishing write tid 2 to smithi16428812-33 2024-04-26T01:07:59.446 INFO:tasks.rados.rados.0.smithi164.stdout:32: done (15 left) 2024-04-26T01:07:59.446 INFO:tasks.rados.rados.0.smithi164.stdout:42: write initial oid 42 2024-04-26T01:07:59.446 INFO:tasks.rados.rados.0.smithi164.stdout:42: seq_num 41 ranges {0=704512} 2024-04-26T01:07:59.447 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:07:59.447 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:07:59.449 INFO:tasks.rados.rados.0.smithi164.stdout:42: writing smithi16428812-42 from 0 to 704512 tid 1 2024-04-26T01:07:59.449 INFO:tasks.rados.rados.0.smithi164.stdout: waiting on 16 2024-04-26T01:07:59.449 INFO:tasks.rados.rados.0.smithi164.stdout:30: finishing write tid 3 to smithi16428812-30 2024-04-26T01:07:59.449 INFO:tasks.rados.rados.0.smithi164.stdout:30: oid 30 updating version 0 to 8 2024-04-26T01:07:59.449 INFO:tasks.rados.rados.0.smithi164.stdout:30: oid 30 version 8 is already newer than 7 2024-04-26T01:07:59.449 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 30 v 8 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-26T01:07:59.449 INFO:tasks.rados.rados.0.smithi164.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-04-26T01:07:59.450 INFO:tasks.rados.rados.0.smithi164.stdout:31: finishing write tid 2 to smithi16428812-31 2024-04-26T01:07:59.450 INFO:tasks.rados.rados.0.smithi164.stdout:30: done (15 left) 2024-04-26T01:07:59.450 INFO:tasks.rados.rados.0.smithi164.stdout:43: write initial oid 43 2024-04-26T01:07:59.450 INFO:tasks.rados.rados.0.smithi164.stdout:43: seq_num 42 ranges {0=720896} 2024-04-26T01:07:59.453 INFO:tasks.rados.rados.0.smithi164.stdout:43: writing smithi16428812-43 from 0 to 720896 tid 1 2024-04-26T01:07:59.454 INFO:tasks.rados.rados.0.smithi164.stdout: waiting on 16 2024-04-26T01:07:59.454 INFO:tasks.rados.rados.0.smithi164.stdout:34: finishing write tid 1 to smithi16428812-34 2024-04-26T01:07:59.454 INFO:tasks.rados.rados.0.smithi164.stdout:34: finishing write tid 2 to smithi16428812-34 2024-04-26T01:07:59.454 INFO:tasks.rados.rados.0.smithi164.stdout:31: finishing write tid 3 to smithi16428812-31 2024-04-26T01:07:59.454 INFO:tasks.rados.rados.0.smithi164.stdout:31: oid 31 updating version 0 to 7 2024-04-26T01:07:59.454 INFO:tasks.rados.rados.0.smithi164.stdout:31: oid 31 updating version 7 to 8 2024-04-26T01:07:59.454 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 31 v 8 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-26T01:07:59.454 INFO:tasks.rados.rados.0.smithi164.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-04-26T01:07:59.454 INFO:tasks.rados.rados.0.smithi164.stdout:33: finishing write tid 3 to smithi16428812-33 2024-04-26T01:07:59.454 INFO:tasks.rados.rados.0.smithi164.stdout:33: oid 33 updating version 0 to 9 2024-04-26T01:07:59.454 INFO:tasks.rados.rados.0.smithi164.stdout:33: oid 33 updating version 9 to 10 2024-04-26T01:07:59.454 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 33 v 10 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-26T01:07:59.454 INFO:tasks.rados.rados.0.smithi164.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-04-26T01:07:59.455 INFO:tasks.rados.rados.0.smithi164.stdout:36: finishing write tid 1 to smithi16428812-36 2024-04-26T01:07:59.455 INFO:tasks.rados.rados.0.smithi164.stdout:27: finishing write tid 1 to smithi16428812-27 2024-04-26T01:07:59.455 INFO:tasks.rados.rados.0.smithi164.stdout:27: finishing write tid 2 to smithi16428812-27 2024-04-26T01:07:59.455 INFO:tasks.rados.rados.0.smithi164.stdout:35: finishing write tid 1 to smithi16428812-35 2024-04-26T01:07:59.455 INFO:tasks.rados.rados.0.smithi164.stdout:36: finishing write tid 2 to smithi16428812-36 2024-04-26T01:07:59.455 INFO:tasks.rados.rados.0.smithi164.stdout:31: done (15 left) 2024-04-26T01:07:59.455 INFO:tasks.rados.rados.0.smithi164.stdout:33: done (14 left) 2024-04-26T01:07:59.455 INFO:tasks.rados.rados.0.smithi164.stdout:44: write initial oid 44 2024-04-26T01:07:59.455 INFO:tasks.rados.rados.0.smithi164.stdout:44: seq_num 43 ranges {0=737280} 2024-04-26T01:07:59.456 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:07:59.449+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:07:59.458 INFO:tasks.rados.rados.0.smithi164.stdout:44: writing smithi16428812-44 from 0 to 737280 tid 1 2024-04-26T01:07:59.458 INFO:tasks.rados.rados.0.smithi164.stdout:45: write initial oid 45 2024-04-26T01:07:59.458 INFO:tasks.rados.rados.0.smithi164.stdout:45: seq_num 44 ranges {0=753664} 2024-04-26T01:07:59.463 INFO:tasks.rados.rados.0.smithi164.stdout:45: writing smithi16428812-45 from 0 to 753664 tid 1 2024-04-26T01:07:59.463 INFO:tasks.rados.rados.0.smithi164.stdout: waiting on 16 2024-04-26T01:07:59.463 INFO:tasks.rados.rados.0.smithi164.stdout:35: finishing write tid 2 to smithi16428812-35 2024-04-26T01:07:59.463 INFO:tasks.rados.rados.0.smithi164.stdout:38: finishing write tid 1 to smithi16428812-38 2024-04-26T01:07:59.463 INFO:tasks.rados.rados.0.smithi164.stdout:37: finishing write tid 1 to smithi16428812-37 2024-04-26T01:07:59.463 INFO:tasks.rados.rados.0.smithi164.stdout:38: finishing write tid 2 to smithi16428812-38 2024-04-26T01:07:59.463 INFO:tasks.rados.rados.0.smithi164.stdout:37: finishing write tid 2 to smithi16428812-37 2024-04-26T01:07:59.463 INFO:tasks.rados.rados.0.smithi164.stdout:36: finishing write tid 3 to smithi16428812-36 2024-04-26T01:07:59.463 INFO:tasks.rados.rados.0.smithi164.stdout:36: oid 36 updating version 0 to 5 2024-04-26T01:07:59.463 INFO:tasks.rados.rados.0.smithi164.stdout:36: oid 36 updating version 5 to 6 2024-04-26T01:07:59.463 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-26T01:07:59.463 INFO:tasks.rados.rados.0.smithi164.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-04-26T01:07:59.463 INFO:tasks.rados.rados.0.smithi164.stdout:35: finishing write tid 3 to smithi16428812-35 2024-04-26T01:07:59.464 INFO:tasks.rados.rados.0.smithi164.stdout:35: oid 35 updating version 0 to 2 2024-04-26T01:07:59.464 INFO:tasks.rados.rados.0.smithi164.stdout:35: oid 35 version 2 is already newer than 1 2024-04-26T01:07:59.464 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 35 v 2 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-26T01:07:59.464 INFO:tasks.rados.rados.0.smithi164.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-04-26T01:07:59.464 INFO:tasks.rados.rados.0.smithi164.stdout:37: finishing write tid 3 to smithi16428812-37 2024-04-26T01:07:59.464 INFO:tasks.rados.rados.0.smithi164.stdout:37: oid 37 updating version 0 to 2 2024-04-26T01:07:59.464 INFO:tasks.rados.rados.0.smithi164.stdout:37: oid 37 version 2 is already newer than 1 2024-04-26T01:07:59.464 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 37 v 2 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-26T01:07:59.464 INFO:tasks.rados.rados.0.smithi164.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-04-26T01:07:59.464 INFO:tasks.rados.rados.0.smithi164.stdout:35: done (15 left) 2024-04-26T01:07:59.464 INFO:tasks.rados.rados.0.smithi164.stdout:36: done (14 left) 2024-04-26T01:07:59.464 INFO:tasks.rados.rados.0.smithi164.stdout:37: done (13 left) 2024-04-26T01:07:59.464 INFO:tasks.rados.rados.0.smithi164.stdout:46: write initial oid 46 2024-04-26T01:07:59.464 INFO:tasks.rados.rados.0.smithi164.stdout:46: seq_num 45 ranges {0=770048} 2024-04-26T01:07:59.467 INFO:tasks.rados.rados.0.smithi164.stdout:46: writing smithi16428812-46 from 0 to 770048 tid 1 2024-04-26T01:07:59.467 INFO:tasks.rados.rados.0.smithi164.stdout:47: write initial oid 47 2024-04-26T01:07:59.467 INFO:tasks.rados.rados.0.smithi164.stdout:47: seq_num 46 ranges {0=786432} 2024-04-26T01:07:59.472 INFO:tasks.rados.rados.0.smithi164.stdout:47: writing smithi16428812-47 from 0 to 786432 tid 1 2024-04-26T01:07:59.472 INFO:tasks.rados.rados.0.smithi164.stdout:48: write initial oid 48 2024-04-26T01:07:59.472 INFO:tasks.rados.rados.0.smithi164.stdout:22: finishing write tid 2 to smithi16428812-22 2024-04-26T01:07:59.472 INFO:tasks.rados.rados.0.smithi164.stdout:34: finishing write tid 3 to smithi16428812-34 2024-04-26T01:07:59.472 INFO:tasks.rados.rados.0.smithi164.stdout:34: oid 34 updating version 0 to 9 2024-04-26T01:07:59.473 INFO:tasks.rados.rados.0.smithi164.stdout:34: oid 34 updating version 9 to 10 2024-04-26T01:07:59.473 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 34 v 10 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-26T01:07:59.473 INFO:tasks.rados.rados.0.smithi164.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-04-26T01:07:59.473 INFO:tasks.rados.rados.0.smithi164.stdout:39: finishing write tid 1 to smithi16428812-39 2024-04-26T01:07:59.473 INFO:tasks.rados.rados.0.smithi164.stdout:39: finishing write tid 2 to smithi16428812-39 2024-04-26T01:07:59.473 INFO:tasks.rados.rados.0.smithi164.stdout:22: finishing write tid 3 to smithi16428812-22 2024-04-26T01:07:59.473 INFO:tasks.rados.rados.0.smithi164.stdout:22: oid 22 updating version 0 to 8 2024-04-26T01:07:59.473 INFO:tasks.rados.rados.0.smithi164.stdout:22: oid 22 version 8 is already newer than 7 2024-04-26T01:07:59.473 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 22 v 8 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-26T01:07:59.473 INFO:tasks.rados.rados.0.smithi164.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-04-26T01:07:59.473 INFO:tasks.rados.rados.0.smithi164.stdout:41: finishing write tid 1 to smithi16428812-41 2024-04-26T01:07:59.473 INFO:tasks.rados.rados.0.smithi164.stdout:40: finishing write tid 1 to smithi16428812-40 2024-04-26T01:07:59.473 INFO:tasks.rados.rados.0.smithi164.stdout:41: finishing write tid 2 to smithi16428812-41 2024-04-26T01:07:59.473 INFO:tasks.rados.rados.0.smithi164.stdout:40: finishing write tid 2 to smithi16428812-40 2024-04-26T01:07:59.473 INFO:tasks.rados.rados.0.smithi164.stdout:42: finishing write tid 1 to smithi16428812-42 2024-04-26T01:07:59.474 INFO:tasks.rados.rados.0.smithi164.stdout:39: finishing write tid 3 to smithi16428812-39 2024-04-26T01:07:59.474 INFO:tasks.rados.rados.0.smithi164.stdout:39: oid 39 updating version 0 to 4 2024-04-26T01:07:59.474 INFO:tasks.rados.rados.0.smithi164.stdout:39: oid 39 version 4 is already newer than 3 2024-04-26T01:07:59.474 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 39 v 4 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-26T01:07:59.474 INFO:tasks.rados.rados.0.smithi164.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-04-26T01:07:59.474 INFO:tasks.rados.rados.0.smithi164.stdout:41: finishing write tid 3 to smithi16428812-41 2024-04-26T01:07:59.474 INFO:tasks.rados.rados.0.smithi164.stdout:41: oid 41 updating version 0 to 6 2024-04-26T01:07:59.474 INFO:tasks.rados.rados.0.smithi164.stdout:41: oid 41 version 6 is already newer than 5 2024-04-26T01:07:59.474 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-26T01:07:59.474 INFO:tasks.rados.rados.0.smithi164.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-04-26T01:07:59.474 INFO:tasks.rados.rados.0.smithi164.stdout:43: finishing write tid 1 to smithi16428812-43 2024-04-26T01:07:59.474 INFO:tasks.rados.rados.0.smithi164.stdout:42: finishing write tid 2 to smithi16428812-42 2024-04-26T01:07:59.474 INFO:tasks.rados.rados.0.smithi164.stdout:40: finishing write tid 3 to smithi16428812-40 2024-04-26T01:07:59.475 INFO:tasks.rados.rados.0.smithi164.stdout:40: oid 40 updating version 0 to 2 2024-04-26T01:07:59.475 INFO:tasks.rados.rados.0.smithi164.stdout:40: oid 40 version 2 is already newer than 1 2024-04-26T01:07:59.475 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 40 v 2 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-26T01:07:59.475 INFO:tasks.rados.rados.0.smithi164.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-04-26T01:07:59.475 INFO:tasks.rados.rados.0.smithi164.stdout:44: finishing write tid 1 to smithi16428812-44 2024-04-26T01:07:59.475 INFO:tasks.rados.rados.0.smithi164.stdout:44: finishing write tid 2 to smithi16428812-44 2024-04-26T01:07:59.475 INFO:tasks.rados.rados.0.smithi164.stdout:42: finishing write tid 3 to smithi16428812-42 2024-04-26T01:07:59.475 INFO:tasks.rados.rados.0.smithi164.stdout:42: oid 42 updating version 0 to 9 2024-04-26T01:07:59.475 INFO:tasks.rados.rados.0.smithi164.stdout:42: oid 42 updating version 9 to 10 2024-04-26T01:07:59.475 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 42 v 10 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-26T01:07:59.475 INFO:tasks.rados.rados.0.smithi164.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-04-26T01:07:59.475 INFO:tasks.rados.rados.0.smithi164.stdout:45: finishing write tid 1 to smithi16428812-45 2024-04-26T01:07:59.475 INFO:tasks.rados.rados.0.smithi164.stdout:21: finishing write tid 2 to smithi16428812-21 2024-04-26T01:07:59.475 INFO:tasks.rados.rados.0.smithi164.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2024-04-26T01:07:59.476 INFO:tasks.rados.rados.0.smithi164.stdout:48: writing smithi16428812-48 from 0 to 425984 tid 1 2024-04-26T01:07:59.477 INFO:tasks.rados.rados.0.smithi164.stdout:48: writing smithi16428812-48 from 425984 to 802816 tid 2 2024-04-26T01:07:59.477 INFO:tasks.rados.rados.0.smithi164.stdout:22: done (15 left) 2024-04-26T01:07:59.477 INFO:tasks.rados.rados.0.smithi164.stdout:34: done (14 left) 2024-04-26T01:07:59.477 INFO:tasks.rados.rados.0.smithi164.stdout:39: done (13 left) 2024-04-26T01:07:59.477 INFO:tasks.rados.rados.0.smithi164.stdout:40: done (12 left) 2024-04-26T01:07:59.477 INFO:tasks.rados.rados.0.smithi164.stdout:41: done (11 left) 2024-04-26T01:07:59.477 INFO:tasks.rados.rados.0.smithi164.stdout:42: done (10 left) 2024-04-26T01:07:59.478 INFO:tasks.rados.rados.0.smithi164.stdout:49: write initial oid 49 2024-04-26T01:07:59.478 INFO:tasks.rados.rados.0.smithi164.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2024-04-26T01:07:59.480 INFO:tasks.rados.rados.0.smithi164.stdout:49: writing smithi16428812-49 from 0 to 442368 tid 1 2024-04-26T01:07:59.481 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:07:59.481 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:07:59.482 INFO:tasks.rados.rados.0.smithi164.stdout:49: writing smithi16428812-49 from 442368 to 819200 tid 2 2024-04-26T01:07:59.482 INFO:tasks.rados.rados.0.smithi164.stdout:50: write initial oid 50 2024-04-26T01:07:59.482 INFO:tasks.rados.rados.0.smithi164.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2024-04-26T01:07:59.485 INFO:tasks.rados.rados.0.smithi164.stdout:50: writing smithi16428812-50 from 0 to 458752 tid 1 2024-04-26T01:07:59.487 INFO:tasks.rados.rados.0.smithi164.stdout:50: writing smithi16428812-50 from 458752 to 835584 tid 2 2024-04-26T01:07:59.487 INFO:tasks.rados.rados.0.smithi164.stdout:51: copy_from oid 42 from oid 34 current snap is 0 2024-04-26T01:07:59.488 INFO:tasks.rados.rados.0.smithi164.stdout:45: finishing write tid 2 to smithi16428812-45 2024-04-26T01:07:59.488 INFO:tasks.rados.rados.0.smithi164.stdout:46: finishing write tid 1 to smithi16428812-46 2024-04-26T01:07:59.488 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 14 current snap is 0 2024-04-26T01:07:59.488 INFO:tasks.rados.rados.0.smithi164.stdout:52: seq_num 50 ranges {229376=475136,704512=376832} 2024-04-26T01:07:59.491 INFO:tasks.rados.rados.0.smithi164.stdout:52: writing smithi16428812-14 from 229376 to 704512 tid 1 2024-04-26T01:07:59.495 INFO:tasks.rados.rados.0.smithi164.stdout:52: writing smithi16428812-14 from 704512 to 1081344 tid 2 2024-04-26T01:07:59.495 INFO:tasks.rados.rados.0.smithi164.stdout:53: read oid 12 snap -1 2024-04-26T01:07:59.495 INFO:tasks.rados.rados.0.smithi164.stdout:53: expect (ObjNum 11 snap 0 seq_num 11) 2024-04-26T01:07:59.495 INFO:tasks.rados.rados.0.smithi164.stdout:54: rmattr oid 15 current snap is 0 2024-04-26T01:07:59.495 INFO:tasks.rados.rados.0.smithi164.stdout:54: done (15 left) 2024-04-26T01:07:59.496 INFO:tasks.rados.rados.0.smithi164.stdout:55: setattr oid 29 current snap is 0 2024-04-26T01:07:59.496 INFO:tasks.rados.rados.0.smithi164.stdout:21: finishing write tid 3 to smithi16428812-21 2024-04-26T01:07:59.496 INFO:tasks.rados.rados.0.smithi164.stdout:21: oid 21 updating version 0 to 4 2024-04-26T01:07:59.496 INFO:tasks.rados.rados.0.smithi164.stdout:21: oid 21 version 4 is already newer than 3 2024-04-26T01:07:59.496 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 21 v 4 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-26T01:07:59.496 INFO:tasks.rados.rados.0.smithi164.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-04-26T01:07:59.496 INFO:tasks.rados.rados.0.smithi164.stdout:45: finishing write tid 3 to smithi16428812-45 2024-04-26T01:07:59.496 INFO:tasks.rados.rados.0.smithi164.stdout:45: oid 45 updating version 0 to 7 2024-04-26T01:07:59.496 INFO:tasks.rados.rados.0.smithi164.stdout:45: oid 45 updating version 7 to 8 2024-04-26T01:07:59.496 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 45 v 8 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-26T01:07:59.496 INFO:tasks.rados.rados.0.smithi164.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-04-26T01:07:59.496 INFO:tasks.rados.rados.0.smithi164.stdout:44: finishing write tid 3 to smithi16428812-44 2024-04-26T01:07:59.496 INFO:tasks.rados.rados.0.smithi164.stdout:44: oid 44 updating version 0 to 3 2024-04-26T01:07:59.496 INFO:tasks.rados.rados.0.smithi164.stdout:44: oid 44 updating version 3 to 4 2024-04-26T01:07:59.497 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 44 v 4 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-26T01:07:59.497 INFO:tasks.rados.rados.0.smithi164.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-04-26T01:07:59.497 INFO:tasks.rados.rados.0.smithi164.stdout:47: finishing write tid 1 to smithi16428812-47 2024-04-26T01:07:59.497 INFO:tasks.rados.rados.0.smithi164.stdout:47: finishing write tid 2 to smithi16428812-47 2024-04-26T01:07:59.497 INFO:tasks.rados.rados.0.smithi164.stdout:48: finishing write tid 1 to smithi16428812-48 2024-04-26T01:07:59.497 INFO:tasks.rados.rados.0.smithi164.stdout:47: finishing write tid 3 to smithi16428812-47 2024-04-26T01:07:59.497 INFO:tasks.rados.rados.0.smithi164.stdout:47: oid 47 updating version 0 to 12 2024-04-26T01:07:59.497 INFO:tasks.rados.rados.0.smithi164.stdout:47: oid 47 version 12 is already newer than 11 2024-04-26T01:07:59.497 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-26T01:07:59.497 INFO:tasks.rados.rados.0.smithi164.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-04-26T01:07:59.497 INFO:tasks.rados.rados.0.smithi164.stdout:49: finishing write tid 1 to smithi16428812-49 2024-04-26T01:07:59.497 INFO:tasks.rados.rados.0.smithi164.stdout:49: finishing write tid 2 to smithi16428812-49 2024-04-26T01:07:59.497 INFO:tasks.rados.rados.0.smithi164.stdout:49: finishing write tid 3 to smithi16428812-49 2024-04-26T01:07:59.497 INFO:tasks.rados.rados.0.smithi164.stdout:48: finishing write tid 2 to smithi16428812-48 2024-04-26T01:07:59.498 INFO:tasks.rados.rados.0.smithi164.stdout:50: finishing write tid 1 to smithi16428812-50 2024-04-26T01:07:59.498 INFO:tasks.rados.rados.0.smithi164.stdout:50: finishing write tid 2 to smithi16428812-50 2024-04-26T01:07:59.498 INFO:tasks.rados.rados.0.smithi164.stdout:49: finishing write tid 4 to smithi16428812-49 2024-04-26T01:07:59.498 INFO:tasks.rados.rados.0.smithi164.stdout:49: oid 49 updating version 0 to 9 2024-04-26T01:07:59.498 INFO:tasks.rados.rados.0.smithi164.stdout:49: oid 49 updating version 9 to 11 2024-04-26T01:07:59.498 INFO:tasks.rados.rados.0.smithi164.stdout:49: oid 49 version 11 is already newer than 10 2024-04-26T01:07:59.498 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 49 v 11 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-26T01:07:59.498 INFO:tasks.rados.rados.0.smithi164.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-04-26T01:07:59.498 INFO:tasks.rados.rados.0.smithi164.stdout:21: done (15 left) 2024-04-26T01:07:59.498 INFO:tasks.rados.rados.0.smithi164.stdout:44: done (14 left) 2024-04-26T01:07:59.498 INFO:tasks.rados.rados.0.smithi164.stdout:45: done (13 left) 2024-04-26T01:07:59.498 INFO:tasks.rados.rados.0.smithi164.stdout:47: done (12 left) 2024-04-26T01:07:59.498 INFO:tasks.rados.rados.0.smithi164.stdout:49: done (11 left) 2024-04-26T01:07:59.498 INFO:tasks.rados.rados.0.smithi164.stdout:56: setattr oid 41 current snap is 0 2024-04-26T01:07:59.499 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 11 current snap is 0 2024-04-26T01:07:59.499 INFO:tasks.rados.rados.0.smithi164.stdout:57: seq_num 51 ranges {180224=491520,671744=376832} 2024-04-26T01:07:59.500 INFO:tasks.rados.rados.0.smithi164.stdout:57: writing smithi16428812-11 from 180224 to 671744 tid 1 2024-04-26T01:07:59.503 INFO:tasks.rados.rados.0.smithi164.stdout:57: writing smithi16428812-11 from 671744 to 1048576 tid 2 2024-04-26T01:07:59.503 INFO:tasks.rados.rados.0.smithi164.stdout:58: setattr oid 49 current snap is 0 2024-04-26T01:07:59.503 INFO:tasks.rados.rados.0.smithi164.stdout:53: expect (ObjNum 11 snap 0 seq_num 11) 2024-04-26T01:07:59.514 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 29 v 7 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-26T01:07:59.514 INFO:tasks.rados.rados.0.smithi164.stdout:48: finishing write tid 3 to smithi16428812-48 2024-04-26T01:07:59.514 INFO:tasks.rados.rados.0.smithi164.stdout:51: finishing copy_from racing read to smithi16428812-42 2024-04-26T01:07:59.514 INFO:tasks.rados.rados.0.smithi164.stdout:50: finishing write tid 3 to smithi16428812-50 2024-04-26T01:07:59.514 INFO:tasks.rados.rados.0.smithi164.stdout:50: finishing write tid 4 to smithi16428812-50 2024-04-26T01:07:59.514 INFO:tasks.rados.rados.0.smithi164.stdout:50: oid 50 updating version 0 to 9 2024-04-26T01:07:59.514 INFO:tasks.rados.rados.0.smithi164.stdout:50: oid 50 version 9 is already newer than 7 2024-04-26T01:07:59.515 INFO:tasks.rados.rados.0.smithi164.stdout:50: oid 50 version 9 is already newer than 8 2024-04-26T01:07:59.515 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-26T01:07:59.515 INFO:tasks.rados.rados.0.smithi164.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-04-26T01:07:59.515 INFO:tasks.rados.rados.0.smithi164.stdout:27: finishing write tid 3 to smithi16428812-27 2024-04-26T01:07:59.515 INFO:tasks.rados.rados.0.smithi164.stdout:27: oid 27 updating version 0 to 6 2024-04-26T01:07:59.515 INFO:tasks.rados.rados.0.smithi164.stdout:27: oid 27 version 6 is already newer than 5 2024-04-26T01:07:59.515 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-26T01:07:59.515 INFO:tasks.rados.rados.0.smithi164.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-04-26T01:07:59.515 INFO:tasks.rados.rados.0.smithi164.stdout:51: finishing copy_from to smithi16428812-42 2024-04-26T01:07:59.515 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 42 v 10 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-26T01:07:59.515 INFO:tasks.rados.rados.0.smithi164.stdout:48: finishing write tid 4 to smithi16428812-48 2024-04-26T01:07:59.515 INFO:tasks.rados.rados.0.smithi164.stdout:48: oid 48 updating version 0 to 6 2024-04-26T01:07:59.515 INFO:tasks.rados.rados.0.smithi164.stdout:48: oid 48 updating version 6 to 7 2024-04-26T01:07:59.515 INFO:tasks.rados.rados.0.smithi164.stdout:48: oid 48 version 7 is already newer than 5 2024-04-26T01:07:59.515 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 48 v 7 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-26T01:07:59.516 INFO:tasks.rados.rados.0.smithi164.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-04-26T01:07:59.516 INFO:tasks.rados.rados.0.smithi164.stdout:52: finishing write tid 1 to smithi16428812-14 2024-04-26T01:07:59.516 INFO:tasks.rados.rados.0.smithi164.stdout:57: finishing write tid 1 to smithi16428812-11 2024-04-26T01:07:59.516 INFO:tasks.rados.rados.0.smithi164.stdout:52: finishing write tid 2 to smithi16428812-14 2024-04-26T01:07:59.516 INFO:tasks.rados.rados.0.smithi164.stdout:52: finishing write tid 3 to smithi16428812-14 2024-04-26T01:07:59.516 INFO:tasks.rados.rados.0.smithi164.stdout:52: finishing write tid 4 to smithi16428812-14 2024-04-26T01:07:59.516 INFO:tasks.rados.rados.0.smithi164.stdout:52: oid 14 updating version 0 to 15 2024-04-26T01:07:59.516 INFO:tasks.rados.rados.0.smithi164.stdout:52: oid 14 version 15 is already newer than 13 2024-04-26T01:07:59.516 INFO:tasks.rados.rados.0.smithi164.stdout:52: oid 14 version 15 is already newer than 14 2024-04-26T01:07:59.516 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 14 v 15 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-04-26T01:07:59.516 INFO:tasks.rados.rados.0.smithi164.stdout:52: left oid 14 (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:07:59.516 INFO:tasks.rados.rados.0.smithi164.stdout:27: done (13 left) 2024-04-26T01:07:59.516 INFO:tasks.rados.rados.0.smithi164.stdout:48: done (12 left) 2024-04-26T01:07:59.516 INFO:tasks.rados.rados.0.smithi164.stdout:50: done (11 left) 2024-04-26T01:07:59.516 INFO:tasks.rados.rados.0.smithi164.stdout:51: done (10 left) 2024-04-26T01:07:59.517 INFO:tasks.rados.rados.0.smithi164.stdout:52: done (9 left) 2024-04-26T01:07:59.517 INFO:tasks.rados.rados.0.smithi164.stdout:53: done (8 left) 2024-04-26T01:07:59.517 INFO:tasks.rados.rados.0.smithi164.stdout:55: done (7 left) 2024-04-26T01:07:59.517 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 21 current snap is 0 2024-04-26T01:07:59.517 INFO:tasks.rados.rados.0.smithi164.stdout:59: seq_num 52 ranges {344064=507904,851968=376832} 2024-04-26T01:07:59.517 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:07:59.517 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:07:59.518 INFO:tasks.rados.rados.0.smithi164.stdout:59: writing smithi16428812-21 from 344064 to 851968 tid 1 2024-04-26T01:07:59.520 INFO:tasks.rados.rados.0.smithi164.stdout:59: writing smithi16428812-21 from 851968 to 1228800 tid 2 2024-04-26T01:07:59.520 INFO:tasks.rados.rados.0.smithi164.stdout:60: delete oid 1 current snap is 0 2024-04-26T01:07:59.521 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-26T01:07:59.524 INFO:tasks.rados.rados.0.smithi164.stdout:46: finishing write tid 2 to smithi16428812-46 2024-04-26T01:07:59.524 INFO:tasks.rados.rados.0.smithi164.stdout:58: done (8 left) 2024-04-26T01:07:59.524 INFO:tasks.rados.rados.0.smithi164.stdout:60: done (7 left) 2024-04-26T01:07:59.524 INFO:tasks.rados.rados.0.smithi164.stdout:61: read oid 34 snap -1 2024-04-26T01:07:59.524 INFO:tasks.rados.rados.0.smithi164.stdout:61: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-26T01:07:59.525 INFO:tasks.rados.rados.0.smithi164.stdout:62: copy_from oid 48 from oid 17 current snap is 0 2024-04-26T01:07:59.525 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 28 current snap is 0 2024-04-26T01:07:59.525 INFO:tasks.rados.rados.0.smithi164.stdout:63: seq_num 54 ranges {458752=540672,999424=376832} 2024-04-26T01:07:59.529 INFO:tasks.rados.rados.0.smithi164.stdout:63: writing smithi16428812-28 from 458752 to 999424 tid 1 2024-04-26T01:07:59.532 INFO:tasks.rados.rados.0.smithi164.stdout:63: writing smithi16428812-28 from 999424 to 1376256 tid 2 2024-04-26T01:07:59.532 INFO:tasks.rados.rados.0.smithi164.stdout:64: rollback oid 33 current snap is 0 2024-04-26T01:07:59.532 INFO:tasks.rados.rados.0.smithi164.stdout:64: done (10 left) 2024-04-26T01:07:59.532 INFO:tasks.rados.rados.0.smithi164.stdout:65: read oid 7 snap -1 2024-04-26T01:07:59.533 INFO:tasks.rados.rados.0.smithi164.stdout:65: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-26T01:07:59.533 INFO:tasks.rados.rados.0.smithi164.stdout:66: delete oid 32 current snap is 0 2024-04-26T01:07:59.533 INFO:tasks.rados.rados.0.smithi164.stdout:59: finishing write tid 1 to smithi16428812-21 2024-04-26T01:07:59.533 INFO:tasks.rados.rados.0.smithi164.stdout:59: finishing write tid 2 to smithi16428812-21 2024-04-26T01:07:59.533 INFO:tasks.rados.rados.0.smithi164.stdout:61: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-26T01:07:59.543 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:07:59.543 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:07:59.555 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:07:59.553+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:07:59.557 INFO:tasks.rados.rados.0.smithi164.stdout:62: finishing copy_from racing read to smithi16428812-48 2024-04-26T01:07:59.557 INFO:tasks.rados.rados.0.smithi164.stdout:65: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-26T01:07:59.562 INFO:tasks.rados.rados.0.smithi164.stdout:43: finishing write tid 2 to smithi16428812-43 2024-04-26T01:07:59.562 INFO:tasks.rados.rados.0.smithi164.stdout:63: finishing write tid 1 to smithi16428812-28 2024-04-26T01:07:59.562 INFO:tasks.rados.rados.0.smithi164.stdout:63: finishing write tid 2 to smithi16428812-28 2024-04-26T01:07:59.563 INFO:tasks.rados.rados.0.smithi164.stdout:63: finishing write tid 3 to smithi16428812-28 2024-04-26T01:07:59.563 INFO:tasks.rados.rados.0.smithi164.stdout:43: finishing write tid 3 to smithi16428812-43 2024-04-26T01:07:59.563 INFO:tasks.rados.rados.0.smithi164.stdout:43: oid 43 updating version 0 to 5 2024-04-26T01:07:59.563 INFO:tasks.rados.rados.0.smithi164.stdout:43: oid 43 updating version 5 to 6 2024-04-26T01:07:59.563 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-26T01:07:59.563 INFO:tasks.rados.rados.0.smithi164.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-04-26T01:07:59.563 INFO:tasks.rados.rados.0.smithi164.stdout:63: finishing write tid 4 to smithi16428812-28 2024-04-26T01:07:59.563 INFO:tasks.rados.rados.0.smithi164.stdout:63: oid 28 updating version 0 to 9 2024-04-26T01:07:59.563 INFO:tasks.rados.rados.0.smithi164.stdout:63: oid 28 version 9 is already newer than 7 2024-04-26T01:07:59.563 INFO:tasks.rados.rados.0.smithi164.stdout:63: oid 28 version 9 is already newer than 8 2024-04-26T01:07:59.563 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 28 v 9 (ObjNum 54 snap 0 seq_num 54) dirty exists 2024-04-26T01:07:59.563 INFO:tasks.rados.rados.0.smithi164.stdout:63: left oid 28 (ObjNum 54 snap 0 seq_num 54) 2024-04-26T01:07:59.563 INFO:tasks.rados.rados.0.smithi164.stdout:43: done (11 left) 2024-04-26T01:07:59.563 INFO:tasks.rados.rados.0.smithi164.stdout:61: done (10 left) 2024-04-26T01:07:59.564 INFO:tasks.rados.rados.0.smithi164.stdout:63: done (9 left) 2024-04-26T01:07:59.564 INFO:tasks.rados.rados.0.smithi164.stdout:65: done (8 left) 2024-04-26T01:07:59.564 INFO:tasks.rados.rados.0.smithi164.stdout:66: done (7 left) 2024-04-26T01:07:59.564 INFO:tasks.rados.rados.0.smithi164.stdout:67: read oid 39 snap -1 2024-04-26T01:07:59.564 INFO:tasks.rados.rados.0.smithi164.stdout:67: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-26T01:07:59.564 INFO:tasks.rados.rados.0.smithi164.stdout:68: delete oid 27 current snap is 0 2024-04-26T01:07:59.564 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:07:59.565 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:07:59.566 INFO:tasks.rados.rados.0.smithi164.stdout:57: finishing write tid 2 to smithi16428812-11 2024-04-26T01:07:59.567 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:07:59.567 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:07:59.567 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:07:59.567 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:07:59.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.567 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.8s4 32/30 ECSubWrite(tid=1, reqid=client.4521.0:8, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:07:59.567 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.312901+0000", 2024-04-26T01:07:59.567 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.25232469400000002, 2024-04-26T01:07:59.567 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0068476539999999999, 2024-04-26T01:07:59.568 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.568 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.568 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:07:59.568 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.568 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.568 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312901+0000", 2024-04-26T01:07:59.568 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.568 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.568 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312901+0000", 2024-04-26T01:07:59.568 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.569 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.569 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312902+0000", 2024-04-26T01:07:59.569 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.381e-06 2024-04-26T01:07:59.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.569 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.569 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312925+0000", 2024-04-26T01:07:59.569 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2489000000000001e-05 2024-04-26T01:07:59.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.569 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.570 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313020+0000", 2024-04-26T01:07:59.570 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.5356e-05 2024-04-26T01:07:59.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.570 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.570 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313031+0000", 2024-04-26T01:07:59.570 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0941000000000001e-05 2024-04-26T01:07:59.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.570 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.570 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313119+0000", 2024-04-26T01:07:59.570 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8381999999999995e-05 2024-04-26T01:07:59.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.571 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:07:59.571 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313135+0000", 2024-04-26T01:07:59.571 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5466999999999999e-05 2024-04-26T01:07:59.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.571 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:07:59.571 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.319701+0000", 2024-04-26T01:07:59.571 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0065662669999999998 2024-04-26T01:07:59.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.571 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.571 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.319748+0000", 2024-04-26T01:07:59.572 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.7370999999999998e-05 2024-04-26T01:07:59.572 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.572 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.572 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.572 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs2 32/30 ECSubWrite(tid=2, reqid=client.4521.0:11, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:07:59.572 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.312975+0000", 2024-04-26T01:07:59.572 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.25225062799999998, 2024-04-26T01:07:59.572 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066966450000000002, 2024-04-26T01:07:59.572 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.572 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.572 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:07:59.572 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.573 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.573 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312975+0000", 2024-04-26T01:07:59.573 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.573 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.573 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312975+0000", 2024-04-26T01:07:59.573 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.573 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.573 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312975+0000", 2024-04-26T01:07:59.573 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.7000000000000004e-07 2024-04-26T01:07:59.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.574 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.574 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312995+0000", 2024-04-26T01:07:59.574 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9613e-05 2024-04-26T01:07:59.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.574 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.574 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313071+0000", 2024-04-26T01:07:59.574 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5945000000000006e-05 2024-04-26T01:07:59.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.574 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.574 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313080+0000", 2024-04-26T01:07:59.574 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5969999999999992e-06 2024-04-26T01:07:59.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.575 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.575 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313173+0000", 2024-04-26T01:07:59.575 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.3579000000000006e-05 2024-04-26T01:07:59.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.575 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:07:59.575 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313184+0000", 2024-04-26T01:07:59.575 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1070000000000001e-05 2024-04-26T01:07:59.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.575 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:07:59.575 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.319628+0000", 2024-04-26T01:07:59.575 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.006443893 2024-04-26T01:07:59.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.576 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.576 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.319671+0000", 2024-04-26T01:07:59.576 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3177999999999999e-05 2024-04-26T01:07:59.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.576 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.576 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.4s5 33/30 ECSubWrite(tid=1, reqid=client.4521.0:16, at_version=0'0, trim_to=0'0, roll_forward_to=32'1))", 2024-04-26T01:07:59.576 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.325537+0000", 2024-04-26T01:07:59.576 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.23968820499999999, 2024-04-26T01:07:59.576 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0053768640000000003, 2024-04-26T01:07:59.576 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.577 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.577 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:07:59.577 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.577 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.577 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325537+0000", 2024-04-26T01:07:59.577 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.577 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.577 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325537+0000", 2024-04-26T01:07:59.577 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.578 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.578 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325538+0000", 2024-04-26T01:07:59.578 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1069999999999999e-06 2024-04-26T01:07:59.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.578 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.578 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325551+0000", 2024-04-26T01:07:59.578 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2557e-05 2024-04-26T01:07:59.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.578 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.578 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325589+0000", 2024-04-26T01:07:59.578 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.782e-05 2024-04-26T01:07:59.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.579 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.579 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.579 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325598+0000", 2024-04-26T01:07:59.579 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7399999999999999e-06 2024-04-26T01:07:59.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.579 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.579 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.579 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325682+0000", 2024-04-26T01:07:59.579 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.4112999999999999e-05 2024-04-26T01:07:59.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.579 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.579 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:07:59.579 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325697+0000", 2024-04-26T01:07:59.579 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4756999999999999e-05 2024-04-26T01:07:59.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.580 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.580 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:07:59.580 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.330862+0000", 2024-04-26T01:07:59.580 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0051649829999999997 2024-04-26T01:07:59.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.580 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.580 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.580 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.330914+0000", 2024-04-26T01:07:59.580 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.1786999999999997e-05 2024-04-26T01:07:59.580 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.580 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.580 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.580 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.581 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.8s4 33/30 ECSubWrite(tid=2, reqid=client.4521.0:8, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:07:59.581 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.326728+0000", 2024-04-26T01:07:59.581 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.238497658, 2024-04-26T01:07:59.581 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0082135509999999995, 2024-04-26T01:07:59.581 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.581 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.581 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:07:59.581 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.581 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.581 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.581 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.326728+0000", 2024-04-26T01:07:59.581 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.581 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.581 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.582 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.326728+0000", 2024-04-26T01:07:59.582 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.582 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.582 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.582 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.326729+0000", 2024-04-26T01:07:59.582 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.5999999999999991e-07 2024-04-26T01:07:59.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.582 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.582 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.582 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.326743+0000", 2024-04-26T01:07:59.582 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.415e-05 2024-04-26T01:07:59.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.583 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.583 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.326788+0000", 2024-04-26T01:07:59.583 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.5123000000000001e-05 2024-04-26T01:07:59.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.583 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.583 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.326797+0000", 2024-04-26T01:07:59.583 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.6309999999999992e-06 2024-04-26T01:07:59.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.583 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.583 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.326872+0000", 2024-04-26T01:07:59.583 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5018000000000006e-05 2024-04-26T01:07:59.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.584 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:07:59.584 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.326882+0000", 2024-04-26T01:07:59.584 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0817e-05 2024-04-26T01:07:59.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.584 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:07:59.584 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.334871+0000", 2024-04-26T01:07:59.584 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079885040000000004 2024-04-26T01:07:59.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.584 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.584 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.334941+0000", 2024-04-26T01:07:59.584 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.0347999999999998e-05 2024-04-26T01:07:59.585 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.585 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.585 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.585 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.ds2 33/30 ECSubWrite(tid=5, reqid=client.4521.0:35, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:07:59.585 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.327769+0000", 2024-04-26T01:07:59.585 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.23745645000000001, 2024-04-26T01:07:59.585 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.007199029, 2024-04-26T01:07:59.585 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.585 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.585 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:07:59.585 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.586 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.586 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327769+0000", 2024-04-26T01:07:59.586 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.586 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.586 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.586 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327769+0000", 2024-04-26T01:07:59.586 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.586 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.586 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.586 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327770+0000", 2024-04-26T01:07:59.586 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.0100000000000004e-07 2024-04-26T01:07:59.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.587 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.587 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327789+0000", 2024-04-26T01:07:59.587 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9046e-05 2024-04-26T01:07:59.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.587 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.587 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327825+0000", 2024-04-26T01:07:59.587 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5834000000000003e-05 2024-04-26T01:07:59.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.587 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.588 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327833+0000", 2024-04-26T01:07:59.588 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7700000000000007e-06 2024-04-26T01:07:59.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.588 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.588 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.588 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327900+0000", 2024-04-26T01:07:59.588 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.6557000000000007e-05 2024-04-26T01:07:59.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.588 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.588 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:07:59.588 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327909+0000", 2024-04-26T01:07:59.588 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.3929999999999997e-06 2024-04-26T01:07:59.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.589 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.589 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:07:59.589 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.334920+0000", 2024-04-26T01:07:59.589 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0070111660000000001 2024-04-26T01:07:59.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.589 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.589 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.589 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.334968+0000", 2024-04-26T01:07:59.589 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.7462000000000001e-05 2024-04-26T01:07:59.589 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.589 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.589 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.589 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.590 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.8s4 33/30 ECSubWrite(tid=4, reqid=client.4521.0:49, at_version=33'3, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:07:59.590 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.346463+0000", 2024-04-26T01:07:59.590 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.218762277, 2024-04-26T01:07:59.590 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.005532258, 2024-04-26T01:07:59.590 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.590 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.590 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:07:59.590 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.590 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.590 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.346463+0000", 2024-04-26T01:07:59.590 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.591 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.591 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.346463+0000", 2024-04-26T01:07:59.591 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.591 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.591 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.346464+0000", 2024-04-26T01:07:59.591 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0219999999999999e-06 2024-04-26T01:07:59.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.591 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.591 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.346609+0000", 2024-04-26T01:07:59.591 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014523800000000001 2024-04-26T01:07:59.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.592 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.592 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.346649+0000", 2024-04-26T01:07:59.592 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.9326999999999998e-05 2024-04-26T01:07:59.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.592 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.592 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.346658+0000", 2024-04-26T01:07:59.592 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.0510000000000002e-06 2024-04-26T01:07:59.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.592 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.592 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.346739+0000", 2024-04-26T01:07:59.592 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.0841999999999996e-05 2024-04-26T01:07:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:07:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.346753+0000", 2024-04-26T01:07:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4695000000000001e-05 2024-04-26T01:07:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:07:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.351949+0000", 2024-04-26T01:07:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0051960460000000002 2024-04-26T01:07:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.351995+0000", 2024-04-26T01:07:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.6037e-05 2024-04-26T01:07:59.594 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.594 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.594 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.594 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.594 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:103 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~573440,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:07:59.594 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.424398+0000", 2024-04-26T01:07:59.594 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.140827428, 2024-04-26T01:07:59.594 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012920365, 2024-04-26T01:07:59.594 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.594 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.594 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:59.594 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:07:59.594 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:07:59.595 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:07:59.595 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 103 2024-04-26T01:07:59.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.595 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.595 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.595 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.424398+0000", 2024-04-26T01:07:59.595 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.595 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.595 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.424398+0000", 2024-04-26T01:07:59.595 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.596 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.596 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.424401+0000", 2024-04-26T01:07:59.596 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0680000000000001e-06 2024-04-26T01:07:59.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.596 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.596 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425367+0000", 2024-04-26T01:07:59.596 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00096614400000000005 2024-04-26T01:07:59.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.596 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.596 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425393+0000", 2024-04-26T01:07:59.596 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.55e-05 2024-04-26T01:07:59.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.597 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.597 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425415+0000", 2024-04-26T01:07:59.597 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1959000000000001e-05 2024-04-26T01:07:59.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.597 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.597 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425510+0000", 2024-04-26T01:07:59.597 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.5134000000000001e-05 2024-04-26T01:07:59.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.597 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:59.597 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425620+0000", 2024-04-26T01:07:59.597 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00011056 2024-04-26T01:07:59.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.598 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.598 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:07:59.598 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.426564+0000", 2024-04-26T01:07:59.598 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00094343200000000002 2024-04-26T01:07:59.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.598 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.598 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:07:59.598 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428445+0000", 2024-04-26T01:07:59.598 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0018816569999999999 2024-04-26T01:07:59.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.598 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.598 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:59.598 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437169+0000", 2024-04-26T01:07:59.599 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0087233220000000004 2024-04-26T01:07:59.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.599 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.599 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437318+0000", 2024-04-26T01:07:59.599 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014958899999999999 2024-04-26T01:07:59.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.599 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.599 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:104 3.0s0 3:0982979f:::smithi16428812-35:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:07:59.599 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.425460+0000", 2024-04-26T01:07:59.599 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.139765586, 2024-04-26T01:07:59.600 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012253794, 2024-04-26T01:07:59.600 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.600 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.600 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:59.600 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:07:59.600 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:07:59.600 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:07:59.600 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 104 2024-04-26T01:07:59.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.600 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.600 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.600 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425460+0000", 2024-04-26T01:07:59.600 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.601 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.601 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425460+0000", 2024-04-26T01:07:59.601 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.601 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.601 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425461+0000", 2024-04-26T01:07:59.601 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7519999999999999e-06 2024-04-26T01:07:59.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.601 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.601 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425470+0000", 2024-04-26T01:07:59.602 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7029999999999995e-06 2024-04-26T01:07:59.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.602 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.602 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425485+0000", 2024-04-26T01:07:59.602 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.472e-05 2024-04-26T01:07:59.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.602 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.602 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425499+0000", 2024-04-26T01:07:59.602 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4228e-05 2024-04-26T01:07:59.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.602 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.603 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.427415+0000", 2024-04-26T01:07:59.603 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001915618 2024-04-26T01:07:59.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.603 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:59.603 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.427541+0000", 2024-04-26T01:07:59.603 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012602699999999999 2024-04-26T01:07:59.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.603 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:07:59.603 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.427948+0000", 2024-04-26T01:07:59.603 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00040687999999999998 2024-04-26T01:07:59.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.604 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:07:59.604 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428977+0000", 2024-04-26T01:07:59.604 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0010294119999999999 2024-04-26T01:07:59.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.604 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.604 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:59.604 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437611+0000", 2024-04-26T01:07:59.604 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0086342789999999999 2024-04-26T01:07:59.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.604 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.604 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.604 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437714+0000", 2024-04-26T01:07:59.604 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000102175 2024-04-26T01:07:59.604 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.604 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.605 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.605 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:105 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:07:59.605 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:07:59.605 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.13970518900000001, 2024-04-26T01:07:59.605 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017139178000000001, 2024-04-26T01:07:59.605 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.605 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.605 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:07:59.605 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:07:59.605 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:07:59.605 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:07:59.605 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 105 2024-04-26T01:07:59.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.606 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.606 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.606 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:07:59.606 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.606 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.606 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:07:59.606 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.606 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.607 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425522+0000", 2024-04-26T01:07:59.607 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5430000000000001e-06 2024-04-26T01:07:59.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.607 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.607 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425529+0000", 2024-04-26T01:07:59.607 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2570000000000001e-06 2024-04-26T01:07:59.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.608 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.608 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425548+0000", 2024-04-26T01:07:59.608 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8674000000000001e-05 2024-04-26T01:07:59.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.608 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.608 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425560+0000", 2024-04-26T01:07:59.608 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2683e-05 2024-04-26T01:07:59.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.608 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.608 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428152+0000", 2024-04-26T01:07:59.608 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0025912800000000001 2024-04-26T01:07:59.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.609 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:07:59.609 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428234+0000", 2024-04-26T01:07:59.609 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2105000000000001e-05 2024-04-26T01:07:59.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.609 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.609 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.609 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438095+0000", 2024-04-26T01:07:59.609 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0098614180000000003 2024-04-26T01:07:59.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.609 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.609 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:59.609 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438205+0000", 2024-04-26T01:07:59.609 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000109849 2024-04-26T01:07:59.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.610 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.610 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.610 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442659+0000", 2024-04-26T01:07:59.610 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0044543689999999997 2024-04-26T01:07:59.610 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.610 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.610 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.610 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.610 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:115 3.0s0 3:00e538ad:::smithi16428812-39:head [append 0~638976,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:07:59.610 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.437822+0000", 2024-04-26T01:07:59.610 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.12740285400000001, 2024-04-26T01:07:59.611 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.013307608, 2024-04-26T01:07:59.611 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.611 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.611 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:59.611 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:07:59.611 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:07:59.611 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:07:59.611 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 115 2024-04-26T01:07:59.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.611 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.611 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.611 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.611 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437822+0000", 2024-04-26T01:07:59.612 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.612 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.612 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437822+0000", 2024-04-26T01:07:59.612 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.612 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.612 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437824+0000", 2024-04-26T01:07:59.612 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.897e-06 2024-04-26T01:07:59.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.612 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.613 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438562+0000", 2024-04-26T01:07:59.613 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00073750400000000005 2024-04-26T01:07:59.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.613 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.613 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438572+0000", 2024-04-26T01:07:59.613 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0570999999999999e-05 2024-04-26T01:07:59.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.613 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.613 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438580+0000", 2024-04-26T01:07:59.613 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.8809999999999996e-06 2024-04-26T01:07:59.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.613 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.614 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.439078+0000", 2024-04-26T01:07:59.614 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00049728399999999996 2024-04-26T01:07:59.614 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.614 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.614 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:59.614 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.439233+0000", 2024-04-26T01:07:59.614 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000155134 2024-04-26T01:07:59.614 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.614 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.614 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:07:59.614 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.440207+0000", 2024-04-26T01:07:59.614 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00097409999999999999 2024-04-26T01:07:59.614 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.614 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.615 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:07:59.615 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442108+0000", 2024-04-26T01:07:59.615 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0019014139999999999 2024-04-26T01:07:59.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.615 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:59.615 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451061+0000", 2024-04-26T01:07:59.615 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0089528560000000004 2024-04-26T01:07:59.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.615 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.615 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451130+0000", 2024-04-26T01:07:59.615 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.8967000000000006e-05 2024-04-26T01:07:59.615 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.615 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.616 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.616 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.616 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:116 3.0s0 3:00e538ad:::smithi16428812-39:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:07:59.616 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.438595+0000", 2024-04-26T01:07:59.616 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.12662995399999999, 2024-04-26T01:07:59.616 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012771675999999999, 2024-04-26T01:07:59.616 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.616 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.616 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:59.616 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:07:59.616 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:07:59.616 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:07:59.616 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 116 2024-04-26T01:07:59.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.617 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.617 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.617 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.617 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438595+0000", 2024-04-26T01:07:59.617 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.617 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.617 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.617 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438595+0000", 2024-04-26T01:07:59.617 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.617 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.617 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.617 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438596+0000", 2024-04-26T01:07:59.618 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7400000000000002e-07 2024-04-26T01:07:59.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.618 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.618 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.618 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438600+0000", 2024-04-26T01:07:59.618 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8039999999999999e-06 2024-04-26T01:07:59.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.618 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.618 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.618 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438607+0000", 2024-04-26T01:07:59.618 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.17e-06 2024-04-26T01:07:59.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.618 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.618 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.618 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438613+0000", 2024-04-26T01:07:59.619 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0549999999999996e-06 2024-04-26T01:07:59.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.619 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.619 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.619 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441123+0000", 2024-04-26T01:07:59.619 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0025094510000000002 2024-04-26T01:07:59.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.619 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.619 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:59.619 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441245+0000", 2024-04-26T01:07:59.619 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012231399999999999 2024-04-26T01:07:59.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.619 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.619 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:07:59.620 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441736+0000", 2024-04-26T01:07:59.620 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00049087600000000001 2024-04-26T01:07:59.620 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.620 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.620 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:07:59.620 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.444541+0000", 2024-04-26T01:07:59.620 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0028048249999999999 2024-04-26T01:07:59.620 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.620 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.620 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:59.620 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451266+0000", 2024-04-26T01:07:59.620 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0067249800000000002 2024-04-26T01:07:59.620 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.620 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.620 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.621 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451367+0000", 2024-04-26T01:07:59.621 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000101327 2024-04-26T01:07:59.621 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.621 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.621 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.621 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.621 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.621 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:117 3.0s0 3:00e538ad:::smithi16428812-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:07:59.621 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:07:59.621 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.126602193, 2024-04-26T01:07:59.621 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017480893000000001, 2024-04-26T01:07:59.621 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.621 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.622 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:07:59.622 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:07:59.622 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:07:59.622 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:07:59.622 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 117 2024-04-26T01:07:59.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.622 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.622 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.622 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.622 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:07:59.622 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.622 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.622 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.623 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:07:59.623 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.623 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.623 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.623 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438624+0000", 2024-04-26T01:07:59.623 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.6000000000000003e-07 2024-04-26T01:07:59.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.623 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.623 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.623 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438628+0000", 2024-04-26T01:07:59.623 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8029999999999998e-06 2024-04-26T01:07:59.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.623 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.624 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.624 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438635+0000", 2024-04-26T01:07:59.624 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2799999999999998e-06 2024-04-26T01:07:59.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.624 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.624 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.624 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438641+0000", 2024-04-26T01:07:59.624 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5459999999999997e-06 2024-04-26T01:07:59.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.624 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.624 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.624 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442203+0000", 2024-04-26T01:07:59.624 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003561594 2024-04-26T01:07:59.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.625 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.625 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:07:59.625 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442287+0000", 2024-04-26T01:07:59.625 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.4443000000000004e-05 2024-04-26T01:07:59.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.625 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.625 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.625 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451430+0000", 2024-04-26T01:07:59.625 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0091427130000000002 2024-04-26T01:07:59.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.625 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.625 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:59.625 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451500+0000", 2024-04-26T01:07:59.626 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9342000000000001e-05 2024-04-26T01:07:59.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.626 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.626 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.626 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.456104+0000", 2024-04-26T01:07:59.626 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0046045119999999998 2024-04-26T01:07:59.626 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.626 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.626 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.626 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.626 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:121 3.0s0 3:0d858d01:::smithi16428812-41:head [append 0~688128,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:07:59.626 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.446304+0000", 2024-04-26T01:07:59.626 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.118921447, 2024-04-26T01:07:59.627 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0088967409999999997, 2024-04-26T01:07:59.627 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.627 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.627 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:59.627 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:07:59.627 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:07:59.627 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:07:59.627 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 121 2024-04-26T01:07:59.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.627 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.627 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.627 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.627 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446304+0000", 2024-04-26T01:07:59.627 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.628 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.628 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.628 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446304+0000", 2024-04-26T01:07:59.628 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.628 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.628 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.628 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446305+0000", 2024-04-26T01:07:59.628 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2529999999999999e-06 2024-04-26T01:07:59.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.628 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.628 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.628 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446941+0000", 2024-04-26T01:07:59.629 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00063629799999999999 2024-04-26T01:07:59.629 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.629 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.629 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.629 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446953+0000", 2024-04-26T01:07:59.629 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1752e-05 2024-04-26T01:07:59.629 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.629 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.629 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.629 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446959+0000", 2024-04-26T01:07:59.629 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.6620000000000002e-06 2024-04-26T01:07:59.629 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.629 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.629 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.630 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446999+0000", 2024-04-26T01:07:59.630 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.9696000000000001e-05 2024-04-26T01:07:59.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.630 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.630 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:59.630 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.447103+0000", 2024-04-26T01:07:59.630 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010429000000000001 2024-04-26T01:07:59.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.630 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.630 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:07:59.630 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.447803+0000", 2024-04-26T01:07:59.630 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00070063500000000002 2024-04-26T01:07:59.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.630 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.631 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:07:59.631 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.449711+0000", 2024-04-26T01:07:59.631 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001907569 2024-04-26T01:07:59.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.631 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.631 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:59.631 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455152+0000", 2024-04-26T01:07:59.631 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0054406630000000001 2024-04-26T01:07:59.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.631 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.631 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.631 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455201+0000", 2024-04-26T01:07:59.631 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.8922999999999999e-05 2024-04-26T01:07:59.631 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.631 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.632 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.632 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:122 3.0s0 3:0d858d01:::smithi16428812-41:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:07:59.632 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.446972+0000", 2024-04-26T01:07:59.632 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.118253636, 2024-04-26T01:07:59.632 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.008358585, 2024-04-26T01:07:59.632 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.632 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.632 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:59.632 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:07:59.632 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:07:59.632 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:07:59.633 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 122 2024-04-26T01:07:59.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.633 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.633 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.633 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.633 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446972+0000", 2024-04-26T01:07:59.633 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.633 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.633 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.633 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446972+0000", 2024-04-26T01:07:59.633 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.633 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.634 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.634 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446972+0000", 2024-04-26T01:07:59.634 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0399999999999996e-07 2024-04-26T01:07:59.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.634 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.634 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.634 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446975+0000", 2024-04-26T01:07:59.634 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.2710000000000001e-06 2024-04-26T01:07:59.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.634 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.634 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.634 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446980+0000", 2024-04-26T01:07:59.634 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.34e-06 2024-04-26T01:07:59.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.635 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.635 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.635 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446983+0000", 2024-04-26T01:07:59.635 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.2140000000000001e-06 2024-04-26T01:07:59.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.635 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.635 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.635 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.448587+0000", 2024-04-26T01:07:59.635 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0016037720000000001 2024-04-26T01:07:59.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.635 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.635 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:59.635 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.448709+0000", 2024-04-26T01:07:59.635 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012216199999999999 2024-04-26T01:07:59.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.636 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.636 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:07:59.636 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.449330+0000", 2024-04-26T01:07:59.636 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00062066400000000005 2024-04-26T01:07:59.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.636 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.636 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:07:59.636 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451871+0000", 2024-04-26T01:07:59.636 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0025411930000000002 2024-04-26T01:07:59.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.636 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.636 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:59.636 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455274+0000", 2024-04-26T01:07:59.636 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003402867 2024-04-26T01:07:59.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.637 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.637 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.637 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455330+0000", 2024-04-26T01:07:59.637 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.6598000000000003e-05 2024-04-26T01:07:59.637 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.637 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.637 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.637 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.637 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:123 3.0s0 3:0d858d01:::smithi16428812-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:07:59.637 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.446995+0000", 2024-04-26T01:07:59.637 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.118230444, 2024-04-26T01:07:59.638 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.01100367, 2024-04-26T01:07:59.638 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.638 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.638 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:07:59.638 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:07:59.638 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:07:59.638 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:07:59.638 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 123 2024-04-26T01:07:59.638 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.638 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.638 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.638 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.638 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446995+0000", 2024-04-26T01:07:59.638 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.639 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.639 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.639 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446995+0000", 2024-04-26T01:07:59.639 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.639 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.639 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.639 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446995+0000", 2024-04-26T01:07:59.639 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4400000000000001e-07 2024-04-26T01:07:59.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.639 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.639 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.640 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446997+0000", 2024-04-26T01:07:59.640 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4160000000000001e-06 2024-04-26T01:07:59.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.640 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.640 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.640 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.447001+0000", 2024-04-26T01:07:59.640 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.4329999999999997e-06 2024-04-26T01:07:59.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.640 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.640 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.640 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.447005+0000", 2024-04-26T01:07:59.640 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.805e-06 2024-04-26T01:07:59.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.640 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.449827+0000", 2024-04-26T01:07:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0028225519999999999 2024-04-26T01:07:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:07:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.449912+0000", 2024-04-26T01:07:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.4765000000000006e-05 2024-04-26T01:07:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455664+0000", 2024-04-26T01:07:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0057516859999999998 2024-04-26T01:07:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455725+0000", 2024-04-26T01:07:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0676e-05 2024-04-26T01:07:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.457999+0000", 2024-04-26T01:07:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0022739930000000002 2024-04-26T01:07:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs2 33/30 ECSubWrite(tid=32, reqid=client.4521.0:134, at_version=33'8, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:07:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.467381+0000", 2024-04-26T01:07:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.097844540999999993, 2024-04-26T01:07:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0052637869999999998, 2024-04-26T01:07:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:07:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467381+0000", 2024-04-26T01:07:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467381+0000", 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467382+0000", 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8599999999999997e-07 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467403+0000", 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1322e-05 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467442+0000", 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8804999999999999e-05 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467452+0000", 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.8810000000000008e-06 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467560+0000", 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010844399999999999 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:07:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467573+0000", 2024-04-26T01:07:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3254e-05 2024-04-26T01:07:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:07:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472590+0000", 2024-04-26T01:07:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.005016904 2024-04-26T01:07:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472645+0000", 2024-04-26T01:07:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.4290999999999999e-05 2024-04-26T01:07:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.647 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.647 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.647 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:150 3.0s0 3:0230bbd3:::smithi16428812-50:head [append 0~458752,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:07:59.647 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.486755+0000", 2024-04-26T01:07:59.647 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.078469993000000002, 2024-04-26T01:07:59.647 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061209979999999999, 2024-04-26T01:07:59.647 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.647 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.647 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:59.647 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:07:59.647 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:07:59.647 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:07:59.647 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 150 2024-04-26T01:07:59.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.648 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.648 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.648 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.648 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.486755+0000", 2024-04-26T01:07:59.648 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.648 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.648 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.648 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.486755+0000", 2024-04-26T01:07:59.648 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.648 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.648 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.648 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.486757+0000", 2024-04-26T01:07:59.649 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1569999999999998e-06 2024-04-26T01:07:59.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.649 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.649 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.487166+0000", 2024-04-26T01:07:59.649 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00040816799999999999 2024-04-26T01:07:59.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.649 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.649 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.487179+0000", 2024-04-26T01:07:59.649 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2977e-05 2024-04-26T01:07:59.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.649 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.649 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.487185+0000", 2024-04-26T01:07:59.650 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.8669999999999999e-06 2024-04-26T01:07:59.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.650 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.650 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.487247+0000", 2024-04-26T01:07:59.650 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.2193999999999994e-05 2024-04-26T01:07:59.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.650 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:59.650 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.487425+0000", 2024-04-26T01:07:59.650 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017843399999999999 2024-04-26T01:07:59.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.651 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:07:59.651 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.488335+0000", 2024-04-26T01:07:59.651 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00090989599999999999 2024-04-26T01:07:59.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.651 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:07:59.651 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.491046+0000", 2024-04-26T01:07:59.651 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0027114560000000001 2024-04-26T01:07:59.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.651 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:59.651 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492771+0000", 2024-04-26T01:07:59.651 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00172489 2024-04-26T01:07:59.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.652 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.652 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.652 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492876+0000", 2024-04-26T01:07:59.652 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000104959 2024-04-26T01:07:59.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.652 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.652 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.652 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:152 3.0s0 3:0230bbd3:::smithi16428812-50:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:07:59.652 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.489313+0000", 2024-04-26T01:07:59.652 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.075912696000000002, 2024-04-26T01:07:59.652 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.015629423, 2024-04-26T01:07:59.652 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.653 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.653 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:07:59.653 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:07:59.653 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:07:59.653 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:07:59.653 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 152 2024-04-26T01:07:59.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.653 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.653 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.653 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.653 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489313+0000", 2024-04-26T01:07:59.653 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.653 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.654 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.654 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489313+0000", 2024-04-26T01:07:59.654 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.654 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.654 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.654 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489314+0000", 2024-04-26T01:07:59.654 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2399999999999996e-07 2024-04-26T01:07:59.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.654 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.654 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.654 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489318+0000", 2024-04-26T01:07:59.654 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3800000000000004e-06 2024-04-26T01:07:59.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.655 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.655 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.655 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489326+0000", 2024-04-26T01:07:59.655 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5459999999999992e-06 2024-04-26T01:07:59.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.655 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.655 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.655 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489335+0000", 2024-04-26T01:07:59.655 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9050000000000003e-06 2024-04-26T01:07:59.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.655 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.655 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.655 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.491271+0000", 2024-04-26T01:07:59.656 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0019354579999999999 2024-04-26T01:07:59.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.656 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.656 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:59.656 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.491416+0000", 2024-04-26T01:07:59.656 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000145396 2024-04-26T01:07:59.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.656 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.656 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:07:59.656 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492139+0000", 2024-04-26T01:07:59.656 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00072277400000000003 2024-04-26T01:07:59.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.656 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.656 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:07:59.656 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492918+0000", 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00077903499999999997 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504860+0000", 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011941712 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504942+0000", 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2293000000000006e-05 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.657 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.658 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:153 3.0s0 3:0230bbd3:::smithi16428812-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:07:59.658 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:07:59.658 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.075879031, 2024-04-26T01:07:59.658 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017766607, 2024-04-26T01:07:59.658 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.658 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.658 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:07:59.658 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:07:59.658 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:07:59.658 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:07:59.658 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 153 2024-04-26T01:07:59.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.658 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.658 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489347+0000", 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9999999999999997e-07 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.659 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489351+0000", 2024-04-26T01:07:59.660 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7309999999999999e-06 2024-04-26T01:07:59.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.660 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.660 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.660 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489359+0000", 2024-04-26T01:07:59.660 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.6619999999999994e-06 2024-04-26T01:07:59.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.660 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.660 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.660 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489366+0000", 2024-04-26T01:07:59.660 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.798e-06 2024-04-26T01:07:59.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.660 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.660 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.660 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492454+0000", 2024-04-26T01:07:59.661 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0030882399999999999 2024-04-26T01:07:59.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.661 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:07:59.661 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492562+0000", 2024-04-26T01:07:59.661 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107948 2024-04-26T01:07:59.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.661 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.661 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504997+0000", 2024-04-26T01:07:59.661 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012434176 2024-04-26T01:07:59.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.661 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:07:59.662 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.505136+0000", 2024-04-26T01:07:59.662 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013994300000000001 2024-04-26T01:07:59.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.662 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.662 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.507113+0000", 2024-04-26T01:07:59.662 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0019764090000000002 2024-04-26T01:07:59.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.662 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.662 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 33/30 ECSubWriteReply(tid=13, last_complete=33'9, committed=1, applied=1))", 2024-04-26T01:07:59.662 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.493174+0000", 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.072051413999999994, 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011776930999999999, 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.493174+0000", 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.493174+0000", 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:07:59.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:07:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.493174+0000", 2024-04-26T01:07:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9699999999999996e-07 2024-04-26T01:07:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:07:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.493177+0000", 2024-04-26T01:07:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0070000000000002e-06 2024-04-26T01:07:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:07:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504751+0000", 2024-04-26T01:07:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011573498999999999 2024-04-26T01:07:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504758+0000", 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7159999999999997e-06 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504810+0000", 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2166000000000002e-05 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504951+0000", 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000141046 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.665 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.666 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.666 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:07:59.666 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:07:59.666 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:07:59.667 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:07:59.667 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:07:59.668 INFO:tasks.rados.rados.0.smithi164.stdout:20: finishing write tid 1 to smithi16428812-20 2024-04-26T01:07:59.668 INFO:tasks.rados.rados.0.smithi164.stdout:67: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-26T01:07:59.668 INFO:tasks.rados.rados.0.smithi164.stdout:20: finishing write tid 2 to smithi16428812-20 2024-04-26T01:07:59.668 INFO:tasks.rados.rados.0.smithi164.stdout:20: finishing write tid 3 to smithi16428812-20 2024-04-26T01:07:59.668 INFO:tasks.rados.rados.0.smithi164.stdout:20: oid 20 updating version 0 to 1 2024-04-26T01:07:59.668 INFO:tasks.rados.rados.0.smithi164.stdout:20: oid 20 updating version 1 to 2 2024-04-26T01:07:59.668 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 20 v 2 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-26T01:07:59.668 INFO:tasks.rados.rados.0.smithi164.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-04-26T01:07:59.668 INFO:tasks.rados.rados.0.smithi164.stdout:46: finishing write tid 3 to smithi16428812-46 2024-04-26T01:07:59.668 INFO:tasks.rados.rados.0.smithi164.stdout:46: oid 46 updating version 0 to 6 2024-04-26T01:07:59.669 INFO:tasks.rados.rados.0.smithi164.stdout:46: oid 46 version 6 is already newer than 5 2024-04-26T01:07:59.669 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-26T01:07:59.669 INFO:tasks.rados.rados.0.smithi164.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-04-26T01:07:59.669 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:07:59.669 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:07:59.670 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:07:59.670 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:07:59.664+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:07:59.671 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:07:59.671 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:07:59.671 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:07:59.671 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:07:59.693 INFO:tasks.rados.rados.0.smithi164.stdout:57: finishing write tid 3 to smithi16428812-11 2024-04-26T01:07:59.697 INFO:tasks.rados.rados.0.smithi164.stdout:20: done (8 left) 2024-04-26T01:07:59.697 INFO:tasks.rados.rados.0.smithi164.stdout:46: done (7 left) 2024-04-26T01:07:59.697 INFO:tasks.rados.rados.0.smithi164.stdout:67: done (6 left) 2024-04-26T01:07:59.697 INFO:tasks.rados.rados.0.smithi164.stdout:68: done (5 left) 2024-04-26T01:07:59.697 INFO:tasks.rados.rados.0.smithi164.stdout:69: rollback oid 9 current snap is 0 2024-04-26T01:07:59.697 INFO:tasks.rados.rados.0.smithi164.stdout:69: done (5 left) 2024-04-26T01:07:59.697 INFO:tasks.rados.rados.0.smithi164.stdout:70: read oid 13 snap -1 2024-04-26T01:07:59.697 INFO:tasks.rados.rados.0.smithi164.stdout:70: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-26T01:07:59.697 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 37 current snap is 0 2024-04-26T01:07:59.697 INFO:tasks.rados.rados.0.smithi164.stdout:71: seq_num 57 ranges {606208=589824,1196032=376832} 2024-04-26T01:07:59.698 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:07:59.698 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:07:59.704 INFO:tasks.rados.rados.0.smithi164.stdout:71: writing smithi16428812-37 from 606208 to 1196032 tid 1 2024-04-26T01:07:59.707 INFO:tasks.rados.rados.0.smithi164.stdout:71: writing smithi16428812-37 from 1196032 to 1572864 tid 2 2024-04-26T01:07:59.707 INFO:tasks.rados.rados.0.smithi164.stdout:72: snap_create 2024-04-26T01:07:59.707 INFO:tasks.rados.rados.0.smithi164.stdout:57: finishing write tid 4 to smithi16428812-11 2024-04-26T01:07:59.707 INFO:tasks.rados.rados.0.smithi164.stdout:57: oid 11 updating version 0 to 9 2024-04-26T01:07:59.707 INFO:tasks.rados.rados.0.smithi164.stdout:57: oid 11 version 9 is already newer than 8 2024-04-26T01:07:59.707 INFO:tasks.rados.rados.0.smithi164.stdout:57: oid 11 version 9 is already newer than 7 2024-04-26T01:07:59.708 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 11 v 9 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-04-26T01:07:59.708 INFO:tasks.rados.rados.0.smithi164.stdout:57: left oid 11 (ObjNum 51 snap 0 seq_num 51) 2024-04-26T01:07:59.708 INFO:tasks.rados.rados.0.smithi164.stdout:62: finishing copy_from to smithi16428812-48 2024-04-26T01:07:59.708 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 48 v 4 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-26T01:07:59.708 INFO:tasks.rados.rados.0.smithi164.stdout:70: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-26T01:07:59.717 INFO:tasks.rados.rados.0.smithi164.stdout:71: finishing write tid 1 to smithi16428812-37 2024-04-26T01:07:59.717 INFO:tasks.rados.rados.0.smithi164.stdout:71: finishing write tid 2 to smithi16428812-37 2024-04-26T01:07:59.728 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:07:59.728 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:07:59.752 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:07:59.752 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:07:59.768 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:07:59.768+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:07:59.779 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:07:59.780 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:07:59.804 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:59.804 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:59.812 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:07:59.812 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:07:59.818 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:07:59.846 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:07:59.846 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:07:59.846 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:07:59.845+0000 7f35d3fff640 0 -- 172.21.15.8:0/1864358597 >> [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] conn(0x7f35d4223230 msgr2=0x7f35d4225620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:07:59.869 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:07:59.865+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:07:59.950 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 41 v 10 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-26T01:07:59.970 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:07:59.968+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:07:59.979 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:07:59.980 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:07:59.993 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:08:00.044 INFO:tasks.rados.rados.0.smithi164.stdout:59: finishing write tid 3 to smithi16428812-21 2024-04-26T01:08:00.048 INFO:tasks.rados.rados.0.smithi164.stdout:59: finishing write tid 4 to smithi16428812-21 2024-04-26T01:08:00.048 INFO:tasks.rados.rados.0.smithi164.stdout:59: oid 21 updating version 0 to 9 2024-04-26T01:08:00.048 INFO:tasks.rados.rados.0.smithi164.stdout:59: oid 21 version 9 is already newer than 8 2024-04-26T01:08:00.048 INFO:tasks.rados.rados.0.smithi164.stdout:59: oid 21 version 9 is already newer than 7 2024-04-26T01:08:00.048 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 21 v 9 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-04-26T01:08:00.048 INFO:tasks.rados.rados.0.smithi164.stdout:59: left oid 21 (ObjNum 52 snap 0 seq_num 52) 2024-04-26T01:08:00.058 INFO:tasks.rados.rados.0.smithi164.stdout:38: finishing write tid 3 to smithi16428812-38 2024-04-26T01:08:00.059 INFO:tasks.rados.rados.0.smithi164.stdout:38: oid 38 updating version 0 to 5 2024-04-26T01:08:00.059 INFO:tasks.rados.rados.0.smithi164.stdout:38: oid 38 updating version 5 to 6 2024-04-26T01:08:00.059 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-26T01:08:00.059 INFO:tasks.rados.rados.0.smithi164.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-04-26T01:08:00.060 INFO:tasks.rados.rados.0.smithi164.stdout:71: finishing write tid 3 to smithi16428812-37 2024-04-26T01:08:00.065 INFO:tasks.rados.rados.0.smithi164.stdout:71: finishing write tid 4 to smithi16428812-37 2024-04-26T01:08:00.065 INFO:tasks.rados.rados.0.smithi164.stdout:71: oid 37 updating version 0 to 4 2024-04-26T01:08:00.065 INFO:tasks.rados.rados.0.smithi164.stdout:71: oid 37 updating version 4 to 5 2024-04-26T01:08:00.065 INFO:tasks.rados.rados.0.smithi164.stdout:71: oid 37 version 5 is already newer than 3 2024-04-26T01:08:00.065 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 37 v 5 (ObjNum 57 snap 0 seq_num 57) dirty exists 2024-04-26T01:08:00.065 INFO:tasks.rados.rados.0.smithi164.stdout:71: left oid 37 (ObjNum 57 snap 0 seq_num 57) 2024-04-26T01:08:00.071 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:00.069+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:00.084 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:00.084 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:00.112 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:00.112 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:00.126 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:08:00.174 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:00.172+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:00.251 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:00.251 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:00.264 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:08:00.273 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:00.269+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:00.311 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-04-26T01:08:00.317 INFO:tasks.rados.rados.0.smithi164.stdout:38: done (7 left) 2024-04-26T01:08:00.317 INFO:tasks.rados.rados.0.smithi164.stdout:56: done (6 left) 2024-04-26T01:08:00.317 INFO:tasks.rados.rados.0.smithi164.stdout:57: done (5 left) 2024-04-26T01:08:00.317 INFO:tasks.rados.rados.0.smithi164.stdout:59: done (4 left) 2024-04-26T01:08:00.318 INFO:tasks.rados.rados.0.smithi164.stdout:62: done (3 left) 2024-04-26T01:08:00.318 INFO:tasks.rados.rados.0.smithi164.stdout:70: done (2 left) 2024-04-26T01:08:00.318 INFO:tasks.rados.rados.0.smithi164.stdout:71: done (1 left) 2024-04-26T01:08:00.318 INFO:tasks.rados.rados.0.smithi164.stdout:72: done (0 left) 2024-04-26T01:08:00.318 INFO:tasks.rados.rados.0.smithi164.stdout:73: rollback oid 2 current snap is 1 2024-04-26T01:08:00.318 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 2 to 0 2024-04-26T01:08:00.318 INFO:tasks.rados.rados.0.smithi164.stdout:74: delete oid 27 current snap is 1 2024-04-26T01:08:00.331 INFO:tasks.rados.rados.0.smithi164.stdout:73: finishing rollback tid 0 to smithi16428812-2 2024-04-26T01:08:00.331 INFO:tasks.rados.rados.0.smithi164.stdout:74: done (1 left) 2024-04-26T01:08:00.331 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 3 current snap is 1 2024-04-26T01:08:00.331 INFO:tasks.rados.rados.0.smithi164.stdout:75: seq_num 59 ranges {49152=622592,671744=376832} 2024-04-26T01:08:00.336 INFO:tasks.rados.rados.0.smithi164.stdout:75: writing smithi16428812-3 from 49152 to 671744 tid 1 2024-04-26T01:08:00.339 INFO:tasks.rados.rados.0.smithi164.stdout:75: writing smithi16428812-3 from 671744 to 1048576 tid 2 2024-04-26T01:08:00.339 INFO:tasks.rados.rados.0.smithi164.stdout:76: setattr oid 24 current snap is 1 2024-04-26T01:08:00.340 INFO:tasks.rados.rados.0.smithi164.stdout:73: finishing rollback tid 1 to smithi16428812-2 2024-04-26T01:08:00.340 INFO:tasks.rados.rados.0.smithi164.stdout:73: finishing rollback tid 2 to smithi16428812-2 2024-04-26T01:08:00.340 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 2 v 10 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-26T01:08:00.342 INFO:tasks.rados.rados.0.smithi164.stdout:73: done (2 left) 2024-04-26T01:08:00.342 INFO:tasks.rados.rados.0.smithi164.stdout:77: rmattr oid 6 current snap is 1 2024-04-26T01:08:00.342 INFO:tasks.rados.rados.0.smithi164.stdout:77: done (2 left) 2024-04-26T01:08:00.342 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 17 current snap is 1 2024-04-26T01:08:00.342 INFO:tasks.rados.rados.0.smithi164.stdout:78: seq_num 60 ranges {278528=638976,917504=376832} 2024-04-26T01:08:00.349 INFO:tasks.rados.rados.0.smithi164.stdout:78: writing smithi16428812-17 from 278528 to 917504 tid 1 2024-04-26T01:08:00.352 INFO:tasks.rados.rados.0.smithi164.stdout:78: writing smithi16428812-17 from 917504 to 1294336 tid 2 2024-04-26T01:08:00.352 INFO:tasks.rados.rados.0.smithi164.stdout:79: snap_create 2024-04-26T01:08:00.352 INFO:tasks.rados.rados.0.smithi164.stdout:75: finishing write tid 1 to smithi16428812-3 2024-04-26T01:08:00.352 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 24 v 8 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-26T01:08:00.353 INFO:tasks.rados.rados.0.smithi164.stdout:75: finishing write tid 2 to smithi16428812-3 2024-04-26T01:08:00.359 INFO:tasks.rados.rados.0.smithi164.stdout:75: finishing write tid 3 to smithi16428812-3 2024-04-26T01:08:00.362 INFO:tasks.rados.rados.0.smithi164.stdout:75: finishing write tid 4 to smithi16428812-3 2024-04-26T01:08:00.362 INFO:tasks.rados.rados.0.smithi164.stdout:75: oid 3 updating version 0 to 8 2024-04-26T01:08:00.362 INFO:tasks.rados.rados.0.smithi164.stdout:75: oid 3 version 8 is already newer than 6 2024-04-26T01:08:00.362 INFO:tasks.rados.rados.0.smithi164.stdout:75: oid 3 version 8 is already newer than 7 2024-04-26T01:08:00.362 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 3 v 8 (ObjNum 59 snap 1 seq_num 59) dirty exists 2024-04-26T01:08:00.362 INFO:tasks.rados.rados.0.smithi164.stdout:75: left oid 3 (ObjNum 59 snap 1 seq_num 59) 2024-04-26T01:08:00.371 INFO:tasks.rados.rados.0.smithi164.stdout:78: finishing write tid 1 to smithi16428812-17 2024-04-26T01:08:00.372 INFO:tasks.rados.rados.0.smithi164.stdout:78: finishing write tid 2 to smithi16428812-17 2024-04-26T01:08:00.372 INFO:tasks.rados.rados.0.smithi164.stdout:78: finishing write tid 3 to smithi16428812-17 2024-04-26T01:08:00.373 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:00.368+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:00.445 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:00.445 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:00.459 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:08:00.502 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:00.504+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:00.574 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:00.569+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:00.626 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:00.626 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:00.640 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:08:00.676 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:00.673+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:00.757 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:00.758 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:00.770 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:08:00.814 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:00.812+0000 7f6879827640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25268) UID: 0 2024-04-26T01:08:00.877 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:00.872+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:00.892 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:00.892 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:00.906 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:08:00.978 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:00.977+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:00.988 INFO:tasks.rados.rados.0.smithi164.stdout:78: finishing write tid 4 to smithi16428812-17 2024-04-26T01:08:00.988 INFO:tasks.rados.rados.0.smithi164.stdout:78: oid 17 updating version 0 to 11 2024-04-26T01:08:00.988 INFO:tasks.rados.rados.0.smithi164.stdout:78: oid 17 version 11 is already newer than 10 2024-04-26T01:08:00.988 INFO:tasks.rados.rados.0.smithi164.stdout:78: oid 17 updating version 11 to 12 2024-04-26T01:08:00.989 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 17 v 12 (ObjNum 60 snap 1 seq_num 60) dirty exists 2024-04-26T01:08:00.989 INFO:tasks.rados.rados.0.smithi164.stdout:78: left oid 17 (ObjNum 60 snap 1 seq_num 60) 2024-04-26T01:08:01.073 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:01.073 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:01.079 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:01.076+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:01.087 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:08:01.180 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:01.177+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:01.252 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:01.252 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:01.266 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:08:01.310 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:01.312+0000 7f6879827640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25268) UID: 0 2024-04-26T01:08:01.324 INFO:tasks.rados.rados.0.smithi164.stdout:75: done (3 left) 2024-04-26T01:08:01.324 INFO:tasks.rados.rados.0.smithi164.stdout:76: done (2 left) 2024-04-26T01:08:01.325 INFO:tasks.rados.rados.0.smithi164.stdout:78: done (1 left) 2024-04-26T01:08:01.325 INFO:tasks.rados.rados.0.smithi164.stdout:79: done (0 left) 2024-04-26T01:08:01.325 INFO:tasks.rados.rados.0.smithi164.stdout:80: read oid 14 snap -1 2024-04-26T01:08:01.325 INFO:tasks.rados.rados.0.smithi164.stdout:80: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:08:01.325 INFO:tasks.rados.rados.0.smithi164.stdout:81: rollback oid 47 current snap is 2 2024-04-26T01:08:01.325 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 47 to 0 2024-04-26T01:08:01.325 INFO:tasks.rados.rados.0.smithi164.stdout:82: read oid 42 snap -1 2024-04-26T01:08:01.325 INFO:tasks.rados.rados.0.smithi164.stdout:82: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-26T01:08:01.325 INFO:tasks.rados.rados.0.smithi164.stdout:83: snap_remove snap 1 2024-04-26T01:08:01.382 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:01.380+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:01.392 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:01.392 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:01.406 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:08:01.482 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:01.480+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:01.573 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:01.573 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:01.583 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:01.577+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:01.586 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:08:01.685 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:01.684+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:01.721 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:01.721 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:01.735 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:08:01.786 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:01.784+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:01.855 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:01.855 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:01.870 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:08:01.887 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:01.884+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:01.988 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:01.988+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:01.998 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:01.999 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:02.013 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:08:02.089 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:02.085+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:02.110 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:08:02.134 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:02.134 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:02.149 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:08:02.190 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:02.192+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:02.279 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:02.279 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:02.292 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:02.292+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:02.307 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:02.307 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:02.312 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:02.312 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:02.328 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:08:02.330 INFO:tasks.rados.rados.0.smithi164.stdout:83: done (3 left) 2024-04-26T01:08:02.330 INFO:tasks.rados.rados.0.smithi164.stdout:84: setattr oid 29 current snap is 2 2024-04-26T01:08:02.330 INFO:tasks.rados.rados.0.smithi164.stdout:80: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:08:02.338 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-26T01:08:02.340 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:02.340 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:02.365 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:02.366 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:02.388 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:02.389 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:02.393 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:02.392+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:02.408 INFO:tasks.rados.rados.0.smithi164.stdout:81: finishing rollback tid 0 to smithi16428812-47 2024-04-26T01:08:02.408 INFO:tasks.rados.rados.0.smithi164.stdout:81: finishing rollback tid 1 to smithi16428812-47 2024-04-26T01:08:02.408 INFO:tasks.rados.rados.0.smithi164.stdout:82: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-26T01:08:02.427 INFO:tasks.rados.rados.0.smithi164.stdout:81: finishing rollback tid 2 to smithi16428812-47 2024-04-26T01:08:02.427 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 47 v 19 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-26T01:08:02.428 INFO:tasks.rados.rados.0.smithi164.stdout:80: done (3 left) 2024-04-26T01:08:02.428 INFO:tasks.rados.rados.0.smithi164.stdout:81: done (2 left) 2024-04-26T01:08:02.428 INFO:tasks.rados.rados.0.smithi164.stdout:82: done (1 left) 2024-04-26T01:08:02.428 INFO:tasks.rados.rados.0.smithi164.stdout:85: delete oid 11 current snap is 2 2024-04-26T01:08:02.431 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:02.431 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:02.438 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 29 v 10 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-26T01:08:02.445 INFO:tasks.rados.rados.0.smithi164.stdout:84: done (1 left) 2024-04-26T01:08:02.445 INFO:tasks.rados.rados.0.smithi164.stdout:85: done (0 left) 2024-04-26T01:08:02.445 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 4 current snap is 2 2024-04-26T01:08:02.445 INFO:tasks.rados.rados.0.smithi164.stdout:86: seq_num 62 ranges {65536=671744,737280=376832} 2024-04-26T01:08:02.456 INFO:tasks.rados.rados.0.smithi164.stdout:86: writing smithi16428812-4 from 65536 to 737280 tid 1 2024-04-26T01:08:02.462 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:02.463 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:02.463 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:02.463 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:02.463 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.463 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:2 3.1s0 3:8bb39a4d:::smithi16428812-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:02.463 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.300456+0000", 2024-04-26T01:08:02.463 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.1582460170000002, 2024-04-26T01:08:02.463 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018845095999999999, 2024-04-26T01:08:02.464 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.464 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.464 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:02.464 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.464 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.464 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.464 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 2 2024-04-26T01:08:02.464 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.464 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.464 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.464 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.464 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300456+0000", 2024-04-26T01:08:02.464 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.464 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.465 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.465 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.465 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300456+0000", 2024-04-26T01:08:02.465 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.465 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.465 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.465 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.465 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300457+0000", 2024-04-26T01:08:02.465 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0999999999999998e-07 2024-04-26T01:08:02.465 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.465 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.465 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.465 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300461+0000", 2024-04-26T01:08:02.466 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0330000000000002e-06 2024-04-26T01:08:02.466 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.466 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.466 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.466 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300468+0000", 2024-04-26T01:08:02.466 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.362e-06 2024-04-26T01:08:02.466 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.466 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.466 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.466 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300474+0000", 2024-04-26T01:08:02.466 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2920000000000003e-06 2024-04-26T01:08:02.466 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.466 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.466 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.467 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302477+0000", 2024-04-26T01:08:02.467 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0020036979999999999 2024-04-26T01:08:02.467 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.467 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.467 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.467 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302595+0000", 2024-04-26T01:08:02.467 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011778900000000001 2024-04-26T01:08:02.467 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.467 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.467 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:02.467 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303203+0000", 2024-04-26T01:08:02.467 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00060815900000000004 2024-04-26T01:08:02.467 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.467 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.467 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307208+0000", 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0040047399999999997 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319228+0000", 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012019556000000001 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319301+0000", 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3856999999999994e-05 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.468 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.469 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.469 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:3 3.1s0 3:8bb39a4d:::smithi16428812-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:02.469 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.300507+0000", 2024-04-26T01:08:02.469 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.1581954740000002, 2024-04-26T01:08:02.469 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.030795241000000001, 2024-04-26T01:08:02.469 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.469 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.469 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:02.469 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.469 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.469 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.469 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 3 2024-04-26T01:08:02.469 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.469 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.470 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.470 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.470 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300507+0000", 2024-04-26T01:08:02.470 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.470 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.470 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.470 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.470 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300507+0000", 2024-04-26T01:08:02.470 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.470 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.470 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.470 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.470 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300507+0000", 2024-04-26T01:08:02.470 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2600000000000002e-07 2024-04-26T01:08:02.470 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.471 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.471 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.471 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300510+0000", 2024-04-26T01:08:02.471 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1940000000000001e-06 2024-04-26T01:08:02.471 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.471 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.471 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.471 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300524+0000", 2024-04-26T01:08:02.471 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4663e-05 2024-04-26T01:08:02.471 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.471 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.471 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.471 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300529+0000", 2024-04-26T01:08:02.471 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8919999999999999e-06 2024-04-26T01:08:02.471 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.472 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.472 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.472 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303652+0000", 2024-04-26T01:08:02.472 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0031224500000000001 2024-04-26T01:08:02.472 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.472 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.472 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:02.472 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303744+0000", 2024-04-26T01:08:02.472 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2109999999999997e-05 2024-04-26T01:08:02.472 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.472 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.472 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.472 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319709+0000", 2024-04-26T01:08:02.472 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015965231 2024-04-26T01:08:02.472 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.473 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.473 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.473 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319786+0000", 2024-04-26T01:08:02.473 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7449999999999999e-05 2024-04-26T01:08:02.473 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.473 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.473 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.473 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.331302+0000", 2024-04-26T01:08:02.473 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011515724999999999 2024-04-26T01:08:02.473 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.473 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.473 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.473 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.473 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.473 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:19 3.7s0 3:e45e30e5:::smithi16428812-7:head [append 0~114688,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:02.474 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.300548+0000", 2024-04-26T01:08:02.474 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.158154025, 2024-04-26T01:08:02.474 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018601116000000001, 2024-04-26T01:08:02.474 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.474 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.474 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:02.474 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.474 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.474 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.474 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 19 2024-04-26T01:08:02.474 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.474 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.474 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.474 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.474 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300548+0000", 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300548+0000", 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300549+0000", 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8999999999999997e-07 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300708+0000", 2024-04-26T01:08:02.475 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015877 2024-04-26T01:08:02.476 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.476 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.476 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.476 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300716+0000", 2024-04-26T01:08:02.476 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7050000000000005e-06 2024-04-26T01:08:02.476 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.476 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.476 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.476 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300724+0000", 2024-04-26T01:08:02.476 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.9240000000000007e-06 2024-04-26T01:08:02.476 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.476 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.476 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.476 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300802+0000", 2024-04-26T01:08:02.477 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7812000000000006e-05 2024-04-26T01:08:02.477 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.477 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.477 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.477 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300958+0000", 2024-04-26T01:08:02.477 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015555000000000001 2024-04-26T01:08:02.477 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.477 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.477 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:02.477 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.301756+0000", 2024-04-26T01:08:02.477 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00079798900000000003 2024-04-26T01:08:02.477 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.477 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.477 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.305686+0000", 2024-04-26T01:08:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0039306769999999996 2024-04-26T01:08:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319067+0000", 2024-04-26T01:08:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013380906 2024-04-26T01:08:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319149+0000", 2024-04-26T01:08:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2293000000000006e-05 2024-04-26T01:08:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:20 3.7s0 3:e45e30e5:::smithi16428812-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.300757+0000", 2024-04-26T01:08:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.1579449639999999, 2024-04-26T01:08:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.01857843, 2024-04-26T01:08:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.480 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.480 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 20 2024-04-26T01:08:02.480 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.480 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.480 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.480 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.480 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300757+0000", 2024-04-26T01:08:02.480 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.480 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.480 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.480 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.480 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300757+0000", 2024-04-26T01:08:02.480 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.480 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.480 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.481 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.481 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300758+0000", 2024-04-26T01:08:02.481 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0500000000000002e-07 2024-04-26T01:08:02.481 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.481 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.481 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.481 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300773+0000", 2024-04-26T01:08:02.481 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4752e-05 2024-04-26T01:08:02.481 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.481 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.481 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.481 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300784+0000", 2024-04-26T01:08:02.481 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1216000000000001e-05 2024-04-26T01:08:02.482 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.482 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.482 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.482 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300799+0000", 2024-04-26T01:08:02.482 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5063e-05 2024-04-26T01:08:02.482 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.482 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.482 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.482 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302445+0000", 2024-04-26T01:08:02.482 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001646123 2024-04-26T01:08:02.482 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.482 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.482 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.482 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302540+0000", 2024-04-26T01:08:02.482 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4400000000000004e-05 2024-04-26T01:08:02.483 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.483 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.483 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:02.483 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302989+0000", 2024-04-26T01:08:02.483 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00044943799999999998 2024-04-26T01:08:02.483 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.483 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.483 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:02.483 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.305690+0000", 2024-04-26T01:08:02.483 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027006259999999998 2024-04-26T01:08:02.483 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.483 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.483 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:02.484 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319255+0000", 2024-04-26T01:08:02.484 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013565315999999999 2024-04-26T01:08:02.484 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.484 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.484 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.484 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319336+0000", 2024-04-26T01:08:02.484 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0691000000000004e-05 2024-04-26T01:08:02.484 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.484 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.484 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.484 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.484 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.484 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:21 3.7s0 3:e45e30e5:::smithi16428812-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:02.484 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.300810+0000", 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.1578919000000001, 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.027297502000000001, 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 21 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300810+0000", 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.485 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.486 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.486 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.486 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300810+0000", 2024-04-26T01:08:02.486 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.486 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.486 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.486 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.486 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300811+0000", 2024-04-26T01:08:02.486 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4600000000000005e-07 2024-04-26T01:08:02.486 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.486 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.486 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.486 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300813+0000", 2024-04-26T01:08:02.486 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4660000000000002e-06 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300819+0000", 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9279999999999998e-06 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300824+0000", 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8690000000000003e-06 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303216+0000", 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002391743 2024-04-26T01:08:02.487 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.488 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.488 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:02.488 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303300+0000", 2024-04-26T01:08:02.488 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3857999999999996e-05 2024-04-26T01:08:02.488 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.488 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.488 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.488 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319712+0000", 2024-04-26T01:08:02.488 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.016412102000000001 2024-04-26T01:08:02.488 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.488 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319825+0000", 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000113062 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.328108+0000", 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0082829280000000002 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:37 3.7s0 3:ed7aa414:::smithi16428812-13:head [append 0~212992,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.321290+0000", 2024-04-26T01:08:02.489 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.1374124600000002, 2024-04-26T01:08:02.490 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015415364000000001, 2024-04-26T01:08:02.490 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.490 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.490 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:02.490 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.490 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.490 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.490 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 37 2024-04-26T01:08:02.490 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.490 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.490 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.490 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.490 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321290+0000", 2024-04-26T01:08:02.490 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.490 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.491 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.491 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.491 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321290+0000", 2024-04-26T01:08:02.491 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.491 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.491 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.491 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.491 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321292+0000", 2024-04-26T01:08:02.491 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6729999999999999e-06 2024-04-26T01:08:02.491 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.491 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.491 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.491 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321615+0000", 2024-04-26T01:08:02.491 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00032335399999999998 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321632+0000", 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7435000000000002e-05 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321644+0000", 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1252e-05 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321788+0000", 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014481300000000001 2024-04-26T01:08:02.492 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.322014+0000", 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00022606699999999999 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.322800+0000", 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00078516999999999997 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.327238+0000", 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0044380390000000004 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.493 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:02.494 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336578+0000", 2024-04-26T01:08:02.494 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0093402680000000005 2024-04-26T01:08:02.494 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.494 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.494 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.494 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336705+0000", 2024-04-26T01:08:02.494 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000127293 2024-04-26T01:08:02.494 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.494 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.494 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.494 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.494 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.494 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:38 3.7s0 3:ed7aa414:::smithi16428812-13:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:02.494 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.321672+0000", 2024-04-26T01:08:02.494 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.137030207, 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.035147350000000001, 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 38 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321672+0000", 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.495 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321672+0000", 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321673+0000", 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0100000000000001e-06 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321680+0000", 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.024e-06 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.496 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.497 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321698+0000", 2024-04-26T01:08:02.497 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8158000000000001e-05 2024-04-26T01:08:02.497 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.497 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.497 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.497 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321707+0000", 2024-04-26T01:08:02.497 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6419999999999996e-06 2024-04-26T01:08:02.497 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.497 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.497 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.497 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325010+0000", 2024-04-26T01:08:02.497 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033028720000000001 2024-04-26T01:08:02.497 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.497 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.498 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.498 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325101+0000", 2024-04-26T01:08:02.498 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1613999999999994e-05 2024-04-26T01:08:02.498 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.498 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.498 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:02.498 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325552+0000", 2024-04-26T01:08:02.498 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00045083800000000001 2024-04-26T01:08:02.498 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.498 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.498 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:02.498 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.328138+0000", 2024-04-26T01:08:02.498 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0025861159999999998 2024-04-26T01:08:02.498 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.498 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.499 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:02.499 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.356713+0000", 2024-04-26T01:08:02.499 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.028574609000000001 2024-04-26T01:08:02.499 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.499 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.499 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.499 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.356819+0000", 2024-04-26T01:08:02.499 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000106467 2024-04-26T01:08:02.499 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.499 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.499 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.499 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.499 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.499 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:39 3.7s0 3:ed7aa414:::smithi16428812-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:02.500 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.321721+0000", 2024-04-26T01:08:02.500 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.136981279, 2024-04-26T01:08:02.500 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.038875445000000002, 2024-04-26T01:08:02.500 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.500 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.500 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:02.500 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.500 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.500 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.500 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 39 2024-04-26T01:08:02.500 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.500 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.500 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.500 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.501 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321721+0000", 2024-04-26T01:08:02.501 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.501 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.501 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.501 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.501 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321721+0000", 2024-04-26T01:08:02.501 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.501 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.501 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.501 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.501 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321722+0000", 2024-04-26T01:08:02.501 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2999999999999999e-07 2024-04-26T01:08:02.501 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.501 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.502 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.502 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321726+0000", 2024-04-26T01:08:02.502 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7639999999999999e-06 2024-04-26T01:08:02.502 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.502 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.502 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.502 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321736+0000", 2024-04-26T01:08:02.502 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0473e-05 2024-04-26T01:08:02.502 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.502 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.502 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.503 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321747+0000", 2024-04-26T01:08:02.503 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0981e-05 2024-04-26T01:08:02.503 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.503 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.503 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.503 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.326352+0000", 2024-04-26T01:08:02.503 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046046239999999999 2024-04-26T01:08:02.503 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.503 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.503 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:02.503 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.326448+0000", 2024-04-26T01:08:02.503 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6668000000000003e-05 2024-04-26T01:08:02.503 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.503 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.503 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.504 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.356889+0000", 2024-04-26T01:08:02.504 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.030441071 2024-04-26T01:08:02.504 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.504 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.504 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.504 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.357378+0000", 2024-04-26T01:08:02.504 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00048859800000000003 2024-04-26T01:08:02.504 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.504 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.504 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.504 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.360597+0000", 2024-04-26T01:08:02.504 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032184359999999999 2024-04-26T01:08:02.504 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.504 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.505 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.505 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.505 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.505 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:65 3.7s0 3:e163c084:::smithi16428812-22:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:02.505 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.364410+0000", 2024-04-26T01:08:02.505 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.0942926640000001, 2024-04-26T01:08:02.505 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.081696361999999995, 2024-04-26T01:08:02.505 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.505 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.505 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:02.505 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.505 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.505 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.505 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 65 2024-04-26T01:08:02.505 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.506 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.506 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.506 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.506 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364410+0000", 2024-04-26T01:08:02.506 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.506 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.506 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.506 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.506 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364410+0000", 2024-04-26T01:08:02.506 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.506 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.506 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.507 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.507 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364411+0000", 2024-04-26T01:08:02.507 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.102e-06 2024-04-26T01:08:02.507 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.507 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.507 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.507 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364416+0000", 2024-04-26T01:08:02.507 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8389999999999997e-06 2024-04-26T01:08:02.507 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.507 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.507 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.507 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364424+0000", 2024-04-26T01:08:02.507 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6100000000000006e-06 2024-04-26T01:08:02.507 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.507 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.508 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.508 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364432+0000", 2024-04-26T01:08:02.508 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5320000000000001e-06 2024-04-26T01:08:02.508 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.508 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.508 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.508 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.365925+0000", 2024-04-26T01:08:02.508 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001493592 2024-04-26T01:08:02.508 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.508 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.508 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.508 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366035+0000", 2024-04-26T01:08:02.508 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000109448 2024-04-26T01:08:02.508 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366490+0000", 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000455427 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.367513+0000", 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001023014 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446008+0000", 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.078494871999999993 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.509 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.510 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.510 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446106+0000", 2024-04-26T01:08:02.510 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7925999999999997e-05 2024-04-26T01:08:02.510 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.510 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.510 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.510 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.510 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.510 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:66 3.7s0 3:e163c084:::smithi16428812-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:02.510 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.364442+0000", 2024-04-26T01:08:02.510 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.0942601359999999, 2024-04-26T01:08:02.510 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.088465429999999998, 2024-04-26T01:08:02.510 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.510 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.510 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 66 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364442+0000", 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364442+0000", 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.511 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364443+0000", 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7899999999999998e-07 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364446+0000", 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9119999999999998e-06 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364454+0000", 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0229999999999992e-06 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.512 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.513 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.513 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364461+0000", 2024-04-26T01:08:02.513 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0110000000000003e-06 2024-04-26T01:08:02.513 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.513 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.513 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.513 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.367245+0000", 2024-04-26T01:08:02.513 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027847499999999999 2024-04-26T01:08:02.513 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.513 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.513 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:02.513 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.367332+0000", 2024-04-26T01:08:02.513 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6118000000000005e-05 2024-04-26T01:08:02.513 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446632+0000", 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.079300689999999993 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446716+0000", 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3391000000000002e-05 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.452908+0000", 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0061919560000000002 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.514 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:78 3.1s0 3:8f91d505:::smithi16428812-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.380021+0000", 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.0786812060000002, 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.023701460000000001, 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 78 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.515 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380021+0000", 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380021+0000", 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380022+0000", 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5700000000000002e-07 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.516 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.517 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380025+0000", 2024-04-26T01:08:02.517 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0900000000000001e-06 2024-04-26T01:08:02.517 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.517 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.517 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.517 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380034+0000", 2024-04-26T01:08:02.517 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1039999999999996e-06 2024-04-26T01:08:02.517 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.517 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.517 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.517 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380040+0000", 2024-04-26T01:08:02.517 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0599999999999996e-06 2024-04-26T01:08:02.517 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.517 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.517 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.383108+0000", 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0030677109999999999 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.383200+0000", 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2689999999999995e-05 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.389150+0000", 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0059497919999999998 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.518 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.389236+0000", 2024-04-26T01:08:02.519 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5784000000000004e-05 2024-04-26T01:08:02.519 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.519 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.519 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.519 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.403723+0000", 2024-04-26T01:08:02.519 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014486572 2024-04-26T01:08:02.519 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.519 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.519 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.519 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.519 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.519 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:92 3.1s0 3:8ab08e97:::smithi16428812-31:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:02.519 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.401379+0000", 2024-04-26T01:08:02.520 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.0573236929999998, 2024-04-26T01:08:02.520 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015990632000000001, 2024-04-26T01:08:02.520 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.520 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.520 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:02.520 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.520 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.520 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.520 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 92 2024-04-26T01:08:02.520 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.520 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.520 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.520 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.520 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401379+0000", 2024-04-26T01:08:02.521 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.521 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.521 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.521 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.521 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401379+0000", 2024-04-26T01:08:02.521 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.521 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.521 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.521 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.521 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401379+0000", 2024-04-26T01:08:02.521 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2999999999999999e-07 2024-04-26T01:08:02.521 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.521 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.521 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.521 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401385+0000", 2024-04-26T01:08:02.522 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.0420000000000002e-06 2024-04-26T01:08:02.522 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.522 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.522 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.522 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401398+0000", 2024-04-26T01:08:02.522 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3018e-05 2024-04-26T01:08:02.522 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.522 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.522 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.522 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401405+0000", 2024-04-26T01:08:02.522 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5410000000000003e-06 2024-04-26T01:08:02.522 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.522 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.522 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.523 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.402965+0000", 2024-04-26T01:08:02.523 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015600849999999999 2024-04-26T01:08:02.523 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.523 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.523 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.523 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.403019+0000", 2024-04-26T01:08:02.523 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3567999999999999e-05 2024-04-26T01:08:02.523 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.523 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.523 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:02.523 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.403397+0000", 2024-04-26T01:08:02.523 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00037848299999999999 2024-04-26T01:08:02.523 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.523 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.524 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:02.524 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.403850+0000", 2024-04-26T01:08:02.524 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00045301400000000001 2024-04-26T01:08:02.524 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.524 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.524 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:02.524 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.417259+0000", 2024-04-26T01:08:02.524 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013409229999999999 2024-04-26T01:08:02.524 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.524 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.524 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.524 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.417369+0000", 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000109821 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:93 3.1s0 3:8ab08e97:::smithi16428812-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.401436+0000", 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.0572663379999998, 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018987277, 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.525 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.526 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 93 2024-04-26T01:08:02.526 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.526 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.526 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.526 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.526 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401436+0000", 2024-04-26T01:08:02.526 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.526 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.526 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.526 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.526 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401436+0000", 2024-04-26T01:08:02.526 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.526 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.526 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.526 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401437+0000", 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0200000000000001e-07 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401440+0000", 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6600000000000001e-06 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401448+0000", 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2110000000000001e-06 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.527 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401452+0000", 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3460000000000004e-06 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.403762+0000", 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0023097880000000001 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.403807+0000", 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.511e-05 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.417495+0000", 2024-04-26T01:08:02.528 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013688117 2024-04-26T01:08:02.529 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.529 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.529 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.529 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.417588+0000", 2024-04-26T01:08:02.529 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3445999999999999e-05 2024-04-26T01:08:02.529 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.529 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.529 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.529 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.420423+0000", 2024-04-26T01:08:02.529 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0028348969999999999 2024-04-26T01:08:02.529 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.529 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.529 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.529 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.529 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:102 3.1s0 3:88871701:::smithi16428812-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.413770+0000", 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.0449324419999999, 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.035191759000000003, 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 102 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.530 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413770+0000", 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413770+0000", 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413770+0000", 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4799999999999998e-07 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413773+0000", 2024-04-26T01:08:02.531 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7499999999999999e-06 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413782+0000", 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7129999999999994e-06 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413791+0000", 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4280000000000002e-06 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.416262+0000", 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024706210000000001 2024-04-26T01:08:02.532 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.533 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.533 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:02.533 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.416358+0000", 2024-04-26T01:08:02.533 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.5775999999999999e-05 2024-04-26T01:08:02.533 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.533 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.533 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.534 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.419853+0000", 2024-04-26T01:08:02.534 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0034948790000000002 2024-04-26T01:08:02.534 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.534 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.534 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.534 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.419904+0000", 2024-04-26T01:08:02.534 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1206999999999999e-05 2024-04-26T01:08:02.534 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.534 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.534 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.534 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.448962+0000", 2024-04-26T01:08:02.534 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.029057837 2024-04-26T01:08:02.534 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.534 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.534 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.535 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.535 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.535 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:120 3.bs0 3:d5691d7a:::smithi16428812-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:02.535 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.443773+0000", 2024-04-26T01:08:02.535 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.0149292509999999, 2024-04-26T01:08:02.535 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.020073483, 2024-04-26T01:08:02.535 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.535 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.535 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:02.535 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.535 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.535 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.535 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 120 2024-04-26T01:08:02.536 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.536 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.536 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.536 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.536 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.443773+0000", 2024-04-26T01:08:02.536 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.536 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.536 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.536 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.536 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.443773+0000", 2024-04-26T01:08:02.536 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.536 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.536 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.536 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.537 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.443774+0000", 2024-04-26T01:08:02.537 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2900000000000003e-07 2024-04-26T01:08:02.537 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.537 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.537 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.537 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.443777+0000", 2024-04-26T01:08:02.537 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1779999999999999e-06 2024-04-26T01:08:02.537 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.537 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.537 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.537 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.443786+0000", 2024-04-26T01:08:02.537 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1959999999999997e-06 2024-04-26T01:08:02.537 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.537 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.538 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.538 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.443794+0000", 2024-04-26T01:08:02.538 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.4800000000000004e-06 2024-04-26T01:08:02.538 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.538 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.538 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.538 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446188+0000", 2024-04-26T01:08:02.538 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002394864 2024-04-26T01:08:02.538 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.538 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.538 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:02.538 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446354+0000", 2024-04-26T01:08:02.538 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000165129 2024-04-26T01:08:02.538 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.538 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.539 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.539 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.454564+0000", 2024-04-26T01:08:02.539 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0082109120000000008 2024-04-26T01:08:02.539 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.539 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.539 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.539 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.454635+0000", 2024-04-26T01:08:02.539 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0937000000000002e-05 2024-04-26T01:08:02.539 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.539 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.539 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.539 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.463847+0000", 2024-04-26T01:08:02.539 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0092111580000000005 2024-04-26T01:08:02.539 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.540 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.540 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.540 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.540 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.540 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:126 3.7s0 3:e8b1d757:::smithi16428812-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:02.540 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.450084+0000", 2024-04-26T01:08:02.540 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.0086184020000002, 2024-04-26T01:08:02.540 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.017170121, 2024-04-26T01:08:02.540 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.540 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.540 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:02.540 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.540 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.540 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.540 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 126 2024-04-26T01:08:02.541 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.541 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.541 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.541 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.541 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.450084+0000", 2024-04-26T01:08:02.541 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.541 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.541 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.541 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.541 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.450084+0000", 2024-04-26T01:08:02.541 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.541 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.541 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.541 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.542 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.450084+0000", 2024-04-26T01:08:02.542 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6499999999999999e-07 2024-04-26T01:08:02.542 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.542 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.542 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.542 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.450087+0000", 2024-04-26T01:08:02.542 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.762e-06 2024-04-26T01:08:02.542 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.542 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.542 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.542 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.450096+0000", 2024-04-26T01:08:02.542 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3469999999999999e-06 2024-04-26T01:08:02.542 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.543 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.543 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.543 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.450102+0000", 2024-04-26T01:08:02.543 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7089999999999996e-06 2024-04-26T01:08:02.543 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.543 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.543 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.543 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.452956+0000", 2024-04-26T01:08:02.543 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0028539379999999999 2024-04-26T01:08:02.543 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.543 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.543 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:02.543 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.453067+0000", 2024-04-26T01:08:02.543 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011068700000000001 2024-04-26T01:08:02.543 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.544 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.544 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.544 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.463761+0000", 2024-04-26T01:08:02.544 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010694416999999999 2024-04-26T01:08:02.544 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.544 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.544 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.544 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.463847+0000", 2024-04-26T01:08:02.544 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5210999999999994e-05 2024-04-26T01:08:02.544 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.544 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.544 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.544 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.467254+0000", 2024-04-26T01:08:02.544 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003407485 2024-04-26T01:08:02.545 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.545 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.545 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.545 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.545 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.545 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:132 3.bs0 3:d071ab05:::smithi16428812-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:02.545 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.460783+0000", 2024-04-26T01:08:02.545 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.9979197289999999, 2024-04-26T01:08:02.545 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.016645619, 2024-04-26T01:08:02.545 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.545 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.545 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:02.545 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.545 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.545 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.546 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 132 2024-04-26T01:08:02.546 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.546 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.546 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.546 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.546 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.460783+0000", 2024-04-26T01:08:02.546 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.546 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.546 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.546 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.546 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.460783+0000", 2024-04-26T01:08:02.546 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.546 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.546 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.546 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.547 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.460783+0000", 2024-04-26T01:08:02.547 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1200000000000003e-07 2024-04-26T01:08:02.547 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.547 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.547 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.547 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.460786+0000", 2024-04-26T01:08:02.547 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8899999999999999e-06 2024-04-26T01:08:02.547 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.547 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.547 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.547 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.460794+0000", 2024-04-26T01:08:02.547 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6429999999999995e-06 2024-04-26T01:08:02.547 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.547 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.547 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.460799+0000", 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5489999999999997e-06 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.463890+0000", 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0030908699999999999 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.463967+0000", 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7032000000000004e-05 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.548 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.466985+0000", 2024-04-26T01:08:02.549 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003018279 2024-04-26T01:08:02.549 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.549 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.549 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.549 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.467057+0000", 2024-04-26T01:08:02.549 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1488000000000004e-05 2024-04-26T01:08:02.549 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.549 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.549 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.549 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477428+0000", 2024-04-26T01:08:02.549 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010371356 2024-04-26T01:08:02.549 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.549 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.549 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.549 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.550 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.550 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:144 3.bs0 3:d8d6d69b:::smithi16428812-48:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:02.550 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.477923+0000", 2024-04-26T01:08:02.550 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.980779428, 2024-04-26T01:08:02.550 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.022386774000000002, 2024-04-26T01:08:02.550 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.550 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.550 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:02.550 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.550 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.550 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.550 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 144 2024-04-26T01:08:02.550 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.550 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.551 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.551 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.551 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477923+0000", 2024-04-26T01:08:02.551 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.551 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.551 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.551 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.551 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477923+0000", 2024-04-26T01:08:02.551 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.551 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.551 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.551 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.551 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477924+0000", 2024-04-26T01:08:02.551 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0200000000000001e-07 2024-04-26T01:08:02.551 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.552 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.552 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.552 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477928+0000", 2024-04-26T01:08:02.552 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.053e-06 2024-04-26T01:08:02.552 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.552 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.552 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.552 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477936+0000", 2024-04-26T01:08:02.552 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5129999999999996e-06 2024-04-26T01:08:02.552 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.552 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.552 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.552 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477942+0000", 2024-04-26T01:08:02.552 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9569999999999999e-06 2024-04-26T01:08:02.552 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.553 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.553 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.553 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.479202+0000", 2024-04-26T01:08:02.553 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0012595600000000001 2024-04-26T01:08:02.553 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.553 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.553 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.553 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.479305+0000", 2024-04-26T01:08:02.553 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000103386 2024-04-26T01:08:02.553 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.553 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.553 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:02.553 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.479724+0000", 2024-04-26T01:08:02.553 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00041885799999999998 2024-04-26T01:08:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.480446+0000", 2024-04-26T01:08:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00072229900000000003 2024-04-26T01:08:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.500215+0000", 2024-04-26T01:08:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.019768626000000001 2024-04-26T01:08:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.500310+0000", 2024-04-26T01:08:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4820000000000004e-05 2024-04-26T01:08:02.555 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.555 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.555 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.555 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.555 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.555 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:145 3.bs0 3:d8d6d69b:::smithi16428812-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:02.555 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.477949+0000", 2024-04-26T01:08:02.555 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.9807534680000001, 2024-04-26T01:08:02.555 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.030723740999999999, 2024-04-26T01:08:02.555 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.555 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.556 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:02.556 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.556 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.556 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.556 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 145 2024-04-26T01:08:02.556 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.556 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.556 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.556 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.556 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477949+0000", 2024-04-26T01:08:02.556 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.556 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.556 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.556 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.556 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477949+0000", 2024-04-26T01:08:02.557 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.557 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.557 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.557 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.557 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477949+0000", 2024-04-26T01:08:02.557 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.5999999999999999e-07 2024-04-26T01:08:02.557 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.557 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.557 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.557 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477951+0000", 2024-04-26T01:08:02.557 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9929999999999998e-06 2024-04-26T01:08:02.557 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.558 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.558 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.558 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477955+0000", 2024-04-26T01:08:02.558 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2760000000000002e-06 2024-04-26T01:08:02.558 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.558 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.558 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.558 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477959+0000", 2024-04-26T01:08:02.558 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7000000000000002e-06 2024-04-26T01:08:02.558 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.558 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.558 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.558 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.480261+0000", 2024-04-26T01:08:02.558 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0023015930000000002 2024-04-26T01:08:02.559 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.559 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.559 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:02.559 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.480344+0000", 2024-04-26T01:08:02.559 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3113000000000002e-05 2024-04-26T01:08:02.559 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.559 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.559 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.559 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.501225+0000", 2024-04-26T01:08:02.559 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.020881591000000001 2024-04-26T01:08:02.559 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.559 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.559 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.560 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.501312+0000", 2024-04-26T01:08:02.560 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6468e-05 2024-04-26T01:08:02.560 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.560 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.560 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.560 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.508673+0000", 2024-04-26T01:08:02.560 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0073606469999999997 2024-04-26T01:08:02.560 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.560 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.560 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.560 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.560 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.560 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:154 3.7s0 3:e8b1d757:::smithi16428812-42:head [copy-from ver 10] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.489221+0000", 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.9694809740000001, 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018704502000000001, 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489221+0000", 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.561 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.562 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.562 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:02.562 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489221+0000", 2024-04-26T01:08:02.562 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:02.562 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.562 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.562 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:02.562 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489223+0000", 2024-04-26T01:08:02.562 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.454e-06 2024-04-26T01:08:02.562 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.562 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.562 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:02.562 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489228+0000", 2024-04-26T01:08:02.562 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3199999999999999e-06 2024-04-26T01:08:02.562 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.563 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.563 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:02.563 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489244+0000", 2024-04-26T01:08:02.563 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5532999999999998e-05 2024-04-26T01:08:02.563 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.563 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.563 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:02.563 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489253+0000", 2024-04-26T01:08:02.563 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2439999999999999e-06 2024-04-26T01:08:02.563 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.563 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.563 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:02.563 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489326+0000", 2024-04-26T01:08:02.563 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3314999999999999e-05 2024-04-26T01:08:02.564 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.564 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.564 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:02.564 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489436+0000", 2024-04-26T01:08:02.564 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010927199999999999 2024-04-26T01:08:02.564 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.564 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.564 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:02.564 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.499328+0000", 2024-04-26T01:08:02.564 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0098929489999999998 2024-04-26T01:08:02.564 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.564 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.564 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:02.564 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.501425+0000", 2024-04-26T01:08:02.564 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0020967799999999999 2024-04-26T01:08:02.565 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.565 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.565 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:02.565 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.507827+0000", 2024-04-26T01:08:02.565 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0064014789999999998 2024-04-26T01:08:02.565 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:02.565 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:02.565 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:02.565 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.507926+0000", 2024-04-26T01:08:02.565 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9155999999999997e-05 2024-04-26T01:08:02.565 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.565 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.565 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.565 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:02.565 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:02.566 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:02.567 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:02.567 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:02.569 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:08:02.569 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:02.570 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:02.570 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:02.570 INFO:tasks.rados.rados.0.smithi164.stdout:86: writing smithi16428812-4 from 737280 to 1114112 tid 2 2024-04-26T01:08:02.571 INFO:tasks.rados.rados.0.smithi164.stdout:87: rollback oid 28 current snap is 2 2024-04-26T01:08:02.571 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 28 to 0 2024-04-26T01:08:02.571 INFO:tasks.rados.rados.0.smithi164.stdout:88: copy_from oid 8 from oid 44 current snap is 2 2024-04-26T01:08:02.571 INFO:tasks.rados.rados.0.smithi164.stdout:89: snap_create 2024-04-26T01:08:02.571 INFO:tasks.rados.rados.0.smithi164.stdout:86: finishing write tid 1 to smithi16428812-4 2024-04-26T01:08:02.571 INFO:tasks.rados.rados.0.smithi164.stdout:86: finishing write tid 2 to smithi16428812-4 2024-04-26T01:08:02.571 INFO:tasks.rados.rados.0.smithi164.stdout:86: finishing write tid 3 to smithi16428812-4 2024-04-26T01:08:02.571 INFO:tasks.rados.rados.0.smithi164.stdout:87: finishing rollback tid 0 to smithi16428812-28 2024-04-26T01:08:02.571 INFO:tasks.rados.rados.0.smithi164.stdout:87: finishing rollback tid 1 to smithi16428812-28 2024-04-26T01:08:02.571 INFO:tasks.rados.rados.0.smithi164.stdout:86: finishing write tid 4 to smithi16428812-4 2024-04-26T01:08:02.571 INFO:tasks.rados.rados.0.smithi164.stdout:86: oid 4 updating version 0 to 14 2024-04-26T01:08:02.571 INFO:tasks.rados.rados.0.smithi164.stdout:86: oid 4 updating version 14 to 16 2024-04-26T01:08:02.571 INFO:tasks.rados.rados.0.smithi164.stdout:86: oid 4 version 16 is already newer than 15 2024-04-26T01:08:02.571 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 4 v 16 (ObjNum 62 snap 2 seq_num 62) dirty exists 2024-04-26T01:08:02.571 INFO:tasks.rados.rados.0.smithi164.stdout:86: left oid 4 (ObjNum 62 snap 2 seq_num 62) 2024-04-26T01:08:02.572 INFO:tasks.rados.rados.0.smithi164.stdout:87: finishing rollback tid 2 to smithi16428812-28 2024-04-26T01:08:02.572 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 28 v 13 (ObjNum 54 snap 0 seq_num 54) dirty exists 2024-04-26T01:08:02.572 INFO:tasks.rados.rados.0.smithi164.stdout:88: finishing copy_from racing read to smithi16428812-8 2024-04-26T01:08:02.572 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:02.572 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:02.572 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:02.573 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:02.573 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:02.573 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:02.561+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:02.598 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:08:02.598 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:08:02.617 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:02.617 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:02.643 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:02.644 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:02.650 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:02.649+0000 7f00bb7fe640 0 -- 172.21.15.8:0/899582225 >> [v2:172.21.15.116:6800/2158650188,v1:172.21.15.116:6801/2158650188] conn(0x7f00bc2212d0 msgr2=0x7f00bc2236c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:08:02.668 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:02.665+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:02.677 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:02.678 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:02.688 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:02.688 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:02.688 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-04-26T01:08:02.688 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:02.702 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:08:02.703 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:02.703 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:02.806 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:02.805+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:02.870 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:02.868+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:02.884 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:02.884 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:02.884 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-04-26T01:08:02.884 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-04-26T01:08:02.885 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:02.899 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:08:02.971 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:02.968+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:03.071 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:03.071 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:03.071 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:03.071 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:03.071 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.071 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.ds5 32/30 ECSubWrite(tid=0, reqid=client.4521.0:4, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:03.071 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.302271+0000", 2024-04-26T01:08:03.071 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.766061224, 2024-04-26T01:08:03.071 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035128360000000001, 2024-04-26T01:08:03.071 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.071 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.072 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.072 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.072 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.072 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.072 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302271+0000", 2024-04-26T01:08:03.072 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.072 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.072 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.072 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.072 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302271+0000", 2024-04-26T01:08:03.072 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.072 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.072 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.073 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.073 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302271+0000", 2024-04-26T01:08:03.073 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3200000000000001e-07 2024-04-26T01:08:03.073 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.073 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.073 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.073 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302330+0000", 2024-04-26T01:08:03.073 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8656999999999998e-05 2024-04-26T01:08:03.073 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.073 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.073 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.073 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302534+0000", 2024-04-26T01:08:03.073 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00020457900000000001 2024-04-26T01:08:03.074 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.074 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.074 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.074 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302550+0000", 2024-04-26T01:08:03.074 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5247e-05 2024-04-26T01:08:03.074 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.074 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.074 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.074 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303551+0000", 2024-04-26T01:08:03.074 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0010011270000000001 2024-04-26T01:08:03.074 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.074 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.074 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.075 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303568+0000", 2024-04-26T01:08:03.075 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7292999999999999e-05 2024-04-26T01:08:03.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.075 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.075 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.305737+0000", 2024-04-26T01:08:03.075 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0021688850000000002 2024-04-26T01:08:03.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.075 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.075 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.305784+0000", 2024-04-26T01:08:03.075 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6715999999999998e-05 2024-04-26T01:08:03.075 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.076 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.076 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.076 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.076 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.076 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.7s4 32/30 ECSubWrite(tid=1, reqid=client.4521.0:19, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:03.076 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.302364+0000", 2024-04-26T01:08:03.076 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.7659678489999999, 2024-04-26T01:08:03.076 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0049735789999999997, 2024-04-26T01:08:03.076 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.076 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.076 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.076 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302364+0000", 2024-04-26T01:08:03.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302364+0000", 2024-04-26T01:08:03.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302364+0000", 2024-04-26T01:08:03.078 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1199999999999999e-07 2024-04-26T01:08:03.078 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.078 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.078 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.078 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302425+0000", 2024-04-26T01:08:03.078 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0485000000000001e-05 2024-04-26T01:08:03.078 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.078 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.078 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.079 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302597+0000", 2024-04-26T01:08:03.079 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017178699999999999 2024-04-26T01:08:03.079 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.079 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.079 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.079 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302607+0000", 2024-04-26T01:08:03.079 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0795000000000001e-05 2024-04-26T01:08:03.079 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.079 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.079 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.079 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303554+0000", 2024-04-26T01:08:03.079 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00094686900000000003 2024-04-26T01:08:03.079 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.079 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.080 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.080 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303564+0000", 2024-04-26T01:08:03.080 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0134e-05 2024-04-26T01:08:03.080 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.080 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.080 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.080 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307301+0000", 2024-04-26T01:08:03.080 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0037365829999999999 2024-04-26T01:08:03.080 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.080 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.080 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.081 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307338+0000", 2024-04-26T01:08:03.081 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6714000000000001e-05 2024-04-26T01:08:03.081 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.081 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.081 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.081 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.081 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.081 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.7s4 32/30 ECSubWrite(tid=2, reqid=client.4521.0:20, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:03.081 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.303606+0000", 2024-04-26T01:08:03.082 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.7647261209999998, 2024-04-26T01:08:03.082 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0045207140000000003, 2024-04-26T01:08:03.082 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.082 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.082 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.082 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.082 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.082 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.082 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303606+0000", 2024-04-26T01:08:03.082 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.082 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.082 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.082 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.083 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303606+0000", 2024-04-26T01:08:03.083 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.083 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.083 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.083 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.083 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303607+0000", 2024-04-26T01:08:03.083 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3859999999999999e-06 2024-04-26T01:08:03.083 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.083 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.083 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.083 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303635+0000", 2024-04-26T01:08:03.083 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8257000000000001e-05 2024-04-26T01:08:03.083 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.083 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.084 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.084 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303695+0000", 2024-04-26T01:08:03.084 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9478000000000002e-05 2024-04-26T01:08:03.084 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.084 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.084 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.084 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303709+0000", 2024-04-26T01:08:03.085 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4375e-05 2024-04-26T01:08:03.085 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.085 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.085 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.085 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.305471+0000", 2024-04-26T01:08:03.085 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0017619770000000001 2024-04-26T01:08:03.086 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.086 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.086 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.086 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.305484+0000", 2024-04-26T01:08:03.086 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2969999999999999e-05 2024-04-26T01:08:03.086 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.086 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.086 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.086 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308086+0000", 2024-04-26T01:08:03.086 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0026017950000000001 2024-04-26T01:08:03.086 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.086 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.086 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.086 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308127+0000", 2024-04-26T01:08:03.087 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0475999999999997e-05 2024-04-26T01:08:03.087 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.087 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.087 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.087 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.087 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.087 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:16 3.4s0 3:2026e263:::smithi16428812-6:head [append 0~98304,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:03.087 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.307585+0000", 2024-04-26T01:08:03.087 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.7607468229999998, 2024-04-26T01:08:03.087 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.016360486, 2024-04-26T01:08:03.087 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.087 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.087 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:03.088 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:03.088 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:03.088 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:03.088 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 16 2024-04-26T01:08:03.088 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.088 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.088 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.088 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.088 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307585+0000", 2024-04-26T01:08:03.088 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.088 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.088 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.088 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.089 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307585+0000", 2024-04-26T01:08:03.089 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.089 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.089 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.089 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.089 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307587+0000", 2024-04-26T01:08:03.089 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1280000000000002e-06 2024-04-26T01:08:03.089 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.089 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.089 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.089 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308008+0000", 2024-04-26T01:08:03.089 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00042080399999999999 2024-04-26T01:08:03.089 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.089 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.090 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.090 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308031+0000", 2024-04-26T01:08:03.090 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3085999999999999e-05 2024-04-26T01:08:03.090 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.090 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.090 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.090 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308041+0000", 2024-04-26T01:08:03.090 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9650000000000003e-06 2024-04-26T01:08:03.090 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.090 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.090 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.090 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308110+0000", 2024-04-26T01:08:03.090 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.915e-05 2024-04-26T01:08:03.091 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.091 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.091 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:03.091 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308277+0000", 2024-04-26T01:08:03.091 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016695800000000001 2024-04-26T01:08:03.091 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.091 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.091 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.091 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.309191+0000", 2024-04-26T01:08:03.091 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00091407599999999999 2024-04-26T01:08:03.091 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.091 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.091 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.091 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314046+0000", 2024-04-26T01:08:03.092 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.004854377 2024-04-26T01:08:03.092 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.092 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.092 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:03.092 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323807+0000", 2024-04-26T01:08:03.092 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0097610869999999999 2024-04-26T01:08:03.092 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.092 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.092 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.092 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323946+0000", 2024-04-26T01:08:03.092 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013885500000000001 2024-04-26T01:08:03.092 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.092 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.093 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.093 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.093 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.093 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es4 33/30 ECSubWrite(tid=0, reqid=client.4521.0:22, at_version=33'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:03.093 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.310807+0000", 2024-04-26T01:08:03.093 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.7575251180000002, 2024-04-26T01:08:03.093 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0040000929999999997, 2024-04-26T01:08:03.093 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.093 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.093 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.093 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.093 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.093 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310807+0000", 2024-04-26T01:08:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310807+0000", 2024-04-26T01:08:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310807+0000", 2024-04-26T01:08:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6899999999999999e-07 2024-04-26T01:08:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310867+0000", 2024-04-26T01:08:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9444999999999998e-05 2024-04-26T01:08:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310964+0000", 2024-04-26T01:08:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7559000000000006e-05 2024-04-26T01:08:03.096 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.096 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.096 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.096 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310976+0000", 2024-04-26T01:08:03.096 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2e-05 2024-04-26T01:08:03.096 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.096 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.096 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.096 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311063+0000", 2024-04-26T01:08:03.096 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7325999999999997e-05 2024-04-26T01:08:03.096 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.096 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.096 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.096 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311075+0000", 2024-04-26T01:08:03.096 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1831000000000001e-05 2024-04-26T01:08:03.097 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.097 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.097 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.097 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314748+0000", 2024-04-26T01:08:03.097 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0036725379999999999 2024-04-26T01:08:03.097 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.097 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.097 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.097 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314807+0000", 2024-04-26T01:08:03.097 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9125000000000001e-05 2024-04-26T01:08:03.097 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.097 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.097 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.097 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.098 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.098 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.ds5 33/30 ECSubWrite(tid=4, reqid=client.4521.0:34, at_version=33'3, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:03.098 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.321813+0000", 2024-04-26T01:08:03.098 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.7465186620000002, 2024-04-26T01:08:03.098 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003286831, 2024-04-26T01:08:03.098 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.098 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.098 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.098 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.098 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.098 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.098 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321813+0000", 2024-04-26T01:08:03.098 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.099 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.099 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.099 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.099 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321813+0000", 2024-04-26T01:08:03.099 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.099 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.099 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.099 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.099 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321814+0000", 2024-04-26T01:08:03.099 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1999999999999998e-07 2024-04-26T01:08:03.099 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.099 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.099 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.100 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.322009+0000", 2024-04-26T01:08:03.100 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019501100000000001 2024-04-26T01:08:03.100 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.100 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.100 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.100 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.322095+0000", 2024-04-26T01:08:03.100 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6297000000000004e-05 2024-04-26T01:08:03.100 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.100 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.100 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.100 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.322109+0000", 2024-04-26T01:08:03.100 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3185e-05 2024-04-26T01:08:03.101 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.101 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.101 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.101 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.322189+0000", 2024-04-26T01:08:03.101 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0573000000000003e-05 2024-04-26T01:08:03.101 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.101 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.101 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.101 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.322202+0000", 2024-04-26T01:08:03.101 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3054000000000001e-05 2024-04-26T01:08:03.101 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.102 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.102 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.102 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325046+0000", 2024-04-26T01:08:03.102 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0028432750000000001 2024-04-26T01:08:03.102 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.102 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.102 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.102 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325100+0000", 2024-04-26T01:08:03.102 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4515999999999998e-05 2024-04-26T01:08:03.102 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.102 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.102 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.102 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.102 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.103 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es4 33/30 ECSubWrite(tid=2, reqid=client.4521.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=33'2))", 2024-04-26T01:08:03.103 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.323329+0000", 2024-04-26T01:08:03.103 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.7450033729999999, 2024-04-26T01:08:03.103 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0037145310000000001, 2024-04-26T01:08:03.103 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.103 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.103 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.103 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.103 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.103 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.103 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323329+0000", 2024-04-26T01:08:03.103 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.103 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.104 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.104 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.104 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323329+0000", 2024-04-26T01:08:03.104 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.104 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.104 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.104 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.104 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323329+0000", 2024-04-26T01:08:03.104 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.03e-07 2024-04-26T01:08:03.104 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.104 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.104 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.104 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323339+0000", 2024-04-26T01:08:03.104 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0069e-05 2024-04-26T01:08:03.104 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.105 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.105 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.105 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323554+0000", 2024-04-26T01:08:03.105 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00021492899999999999 2024-04-26T01:08:03.105 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.105 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.105 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.105 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323561+0000", 2024-04-26T01:08:03.105 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2679999999999997e-06 2024-04-26T01:08:03.105 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.105 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.105 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.105 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323710+0000", 2024-04-26T01:08:03.106 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014928399999999999 2024-04-26T01:08:03.106 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.106 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.106 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.106 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323723+0000", 2024-04-26T01:08:03.106 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2336e-05 2024-04-26T01:08:03.106 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.106 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.106 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.106 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.326991+0000", 2024-04-26T01:08:03.106 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032684469999999998 2024-04-26T01:08:03.106 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.107 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.107 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.107 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.327043+0000", 2024-04-26T01:08:03.107 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1894999999999997e-05 2024-04-26T01:08:03.107 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.107 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.107 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.107 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.107 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.107 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.7s4 33/30 ECSubWrite(tid=8, reqid=client.4521.0:37, at_version=33'3, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:03.107 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.323408+0000", 2024-04-26T01:08:03.107 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.7449237589999997, 2024-04-26T01:08:03.107 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003773476, 2024-04-26T01:08:03.108 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.108 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.108 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.108 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.108 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.108 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.108 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323408+0000", 2024-04-26T01:08:03.108 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.108 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.108 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.108 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.108 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323408+0000", 2024-04-26T01:08:03.109 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.109 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.109 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.109 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.109 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323409+0000", 2024-04-26T01:08:03.109 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.012e-06 2024-04-26T01:08:03.109 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.109 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.109 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.109 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323489+0000", 2024-04-26T01:08:03.109 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.9507999999999998e-05 2024-04-26T01:08:03.109 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.109 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.109 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.110 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323600+0000", 2024-04-26T01:08:03.110 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000110927 2024-04-26T01:08:03.110 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.110 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.110 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.110 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323610+0000", 2024-04-26T01:08:03.110 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0478e-05 2024-04-26T01:08:03.110 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.110 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.110 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.110 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323795+0000", 2024-04-26T01:08:03.110 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00018458400000000001 2024-04-26T01:08:03.110 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.110 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.111 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.111 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323806+0000", 2024-04-26T01:08:03.111 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1385e-05 2024-04-26T01:08:03.111 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.111 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.111 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.111 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.327152+0000", 2024-04-26T01:08:03.111 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033455590000000001 2024-04-26T01:08:03.111 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.111 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.111 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.111 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.327182+0000", 2024-04-26T01:08:03.111 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0023e-05 2024-04-26T01:08:03.111 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.112 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.112 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.112 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.112 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.112 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:17 3.4s0 3:2026e263:::smithi16428812-6:head [setxattr _header (55) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:03.112 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.335986+0000", 2024-04-26T01:08:03.112 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.732345617, 2024-04-26T01:08:03.112 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010264923, 2024-04-26T01:08:03.112 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.112 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.112 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:03.112 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:03.112 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:03.112 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:03.112 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 17 2024-04-26T01:08:03.113 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.113 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.113 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.113 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.113 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.335986+0000", 2024-04-26T01:08:03.113 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.113 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.113 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.113 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.113 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.335986+0000", 2024-04-26T01:08:03.113 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.113 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.113 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.114 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.114 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.335988+0000", 2024-04-26T01:08:03.114 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.373e-06 2024-04-26T01:08:03.114 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.114 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.114 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.114 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.335994+0000", 2024-04-26T01:08:03.114 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0700000000000003e-06 2024-04-26T01:08:03.114 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.114 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.114 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.114 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336007+0000", 2024-04-26T01:08:03.114 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3577000000000001e-05 2024-04-26T01:08:03.114 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.115 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.115 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.115 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336018+0000", 2024-04-26T01:08:03.115 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0668e-05 2024-04-26T01:08:03.115 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.115 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.115 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.115 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336285+0000", 2024-04-26T01:08:03.115 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026682799999999999 2024-04-26T01:08:03.115 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.115 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.115 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:03.115 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336439+0000", 2024-04-26T01:08:03.115 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000154048 2024-04-26T01:08:03.116 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.116 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.116 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.116 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.338115+0000", 2024-04-26T01:08:03.116 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0016757880000000001 2024-04-26T01:08:03.116 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.116 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.116 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.116 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.339187+0000", 2024-04-26T01:08:03.116 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001072494 2024-04-26T01:08:03.116 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.116 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.116 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:03.116 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.346146+0000", 2024-04-26T01:08:03.117 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0069590279999999999 2024-04-26T01:08:03.117 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.117 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.117 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.117 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.346251+0000", 2024-04-26T01:08:03.117 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000105049 2024-04-26T01:08:03.117 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.117 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.117 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.117 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.117 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.117 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:18 3.4s0 3:2026e263:::smithi16428812-6:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:03.117 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.336046+0000", 2024-04-26T01:08:03.117 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.7322856930000001, 2024-04-26T01:08:03.117 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013179965, 2024-04-26T01:08:03.118 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.118 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.118 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:03.118 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:03.118 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:03.118 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:03.118 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 18 2024-04-26T01:08:03.118 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.118 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.118 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.118 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.118 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336046+0000", 2024-04-26T01:08:03.118 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.119 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.119 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.119 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.119 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336046+0000", 2024-04-26T01:08:03.119 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.119 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.119 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.119 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.119 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336047+0000", 2024-04-26T01:08:03.119 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1119999999999999e-06 2024-04-26T01:08:03.119 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.119 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.119 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.119 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336052+0000", 2024-04-26T01:08:03.120 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3510000000000004e-06 2024-04-26T01:08:03.120 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.120 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.120 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.120 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336063+0000", 2024-04-26T01:08:03.120 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1722999999999999e-05 2024-04-26T01:08:03.120 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.120 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.120 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.120 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336070+0000", 2024-04-26T01:08:03.120 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6259999999999997e-06 2024-04-26T01:08:03.120 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.120 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.121 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.121 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.338893+0000", 2024-04-26T01:08:03.121 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0028230009999999999 2024-04-26T01:08:03.121 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.121 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.121 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:03.121 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.339060+0000", 2024-04-26T01:08:03.121 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016734700000000001 2024-04-26T01:08:03.121 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.121 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.121 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.121 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.346619+0000", 2024-04-26T01:08:03.121 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0075586949999999998 2024-04-26T01:08:03.121 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.122 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.122 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:03.122 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.346731+0000", 2024-04-26T01:08:03.122 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000111863 2024-04-26T01:08:03.122 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.122 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.122 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.122 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349226+0000", 2024-04-26T01:08:03.122 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024952469999999999 2024-04-26T01:08:03.122 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.122 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.122 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.123 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.123 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.123 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:85 3.4s0 3:2d34bde3:::smithi16428812-29:head [append 0~475136,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:03.123 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.391877+0000", 2024-04-26T01:08:03.123 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.6764554020000002, 2024-04-26T01:08:03.123 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0061425170000000001, 2024-04-26T01:08:03.123 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.123 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.123 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:03.123 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:03.123 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:03.123 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:03.123 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 85 2024-04-26T01:08:03.123 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.124 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.124 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.124 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.124 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.391877+0000", 2024-04-26T01:08:03.124 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.124 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.124 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.124 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.124 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.391877+0000", 2024-04-26T01:08:03.124 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.124 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.124 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.124 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.125 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.391879+0000", 2024-04-26T01:08:03.125 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1210000000000001e-06 2024-04-26T01:08:03.125 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.125 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.125 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.125 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392791+0000", 2024-04-26T01:08:03.125 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00091226600000000001 2024-04-26T01:08:03.125 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.125 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.125 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.125 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392845+0000", 2024-04-26T01:08:03.125 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4191999999999999e-05 2024-04-26T01:08:03.125 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.125 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.126 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.126 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392855+0000", 2024-04-26T01:08:03.126 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0176e-05 2024-04-26T01:08:03.126 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.126 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.126 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.126 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392919+0000", 2024-04-26T01:08:03.126 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3633000000000006e-05 2024-04-26T01:08:03.126 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.126 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.126 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:03.126 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.393094+0000", 2024-04-26T01:08:03.126 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017526099999999999 2024-04-26T01:08:03.127 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.127 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.127 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.127 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.394178+0000", 2024-04-26T01:08:03.127 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0010841010000000001 2024-04-26T01:08:03.127 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.127 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.127 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.127 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.396024+0000", 2024-04-26T01:08:03.127 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0018452950000000001 2024-04-26T01:08:03.127 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.127 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.127 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:03.128 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.397770+0000", 2024-04-26T01:08:03.128 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0017461449999999999 2024-04-26T01:08:03.128 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.128 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.128 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.128 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398019+0000", 2024-04-26T01:08:03.128 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00024932699999999998 2024-04-26T01:08:03.128 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.128 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.128 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.128 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.128 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.128 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:86 3.4s0 3:2d34bde3:::smithi16428812-29:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:03.128 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.392871+0000", 2024-04-26T01:08:03.129 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.675460481, 2024-04-26T01:08:03.129 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0059468639999999996, 2024-04-26T01:08:03.129 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.129 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.129 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:03.129 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:03.129 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:03.129 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:03.129 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 86 2024-04-26T01:08:03.129 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.129 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.129 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.129 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.130 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392871+0000", 2024-04-26T01:08:03.130 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.130 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.130 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.130 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.130 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392871+0000", 2024-04-26T01:08:03.130 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.130 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.130 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.130 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.130 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392872+0000", 2024-04-26T01:08:03.130 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7000000000000003e-07 2024-04-26T01:08:03.130 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.131 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.131 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.131 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392879+0000", 2024-04-26T01:08:03.131 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4620000000000001e-06 2024-04-26T01:08:03.131 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.131 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.131 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.131 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392888+0000", 2024-04-26T01:08:03.131 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1929999999999999e-06 2024-04-26T01:08:03.131 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.131 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.131 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.131 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392893+0000", 2024-04-26T01:08:03.131 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4410000000000001e-06 2024-04-26T01:08:03.132 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.132 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.132 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.132 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.394996+0000", 2024-04-26T01:08:03.132 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0021021310000000001 2024-04-26T01:08:03.132 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.132 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.132 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:03.132 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.395096+0000", 2024-04-26T01:08:03.132 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000100352 2024-04-26T01:08:03.132 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.132 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.132 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.132 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.395564+0000", 2024-04-26T01:08:03.133 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00046836000000000002 2024-04-26T01:08:03.133 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.133 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.133 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.133 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.396213+0000", 2024-04-26T01:08:03.133 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00064827299999999999 2024-04-26T01:08:03.133 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.133 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.133 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:03.133 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398676+0000", 2024-04-26T01:08:03.133 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024638279999999999 2024-04-26T01:08:03.133 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.133 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.133 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.134 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398818+0000", 2024-04-26T01:08:03.134 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014185399999999999 2024-04-26T01:08:03.134 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.134 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.134 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.134 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.134 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.134 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:87 3.4s0 3:2d34bde3:::smithi16428812-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:03.134 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.392902+0000", 2024-04-26T01:08:03.134 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.6754294559999998, 2024-04-26T01:08:03.134 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010089598999999999, 2024-04-26T01:08:03.134 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.134 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.134 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:03.135 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:03.135 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:03.135 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:03.135 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 87 2024-04-26T01:08:03.135 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.135 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.135 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.135 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.135 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392902+0000", 2024-04-26T01:08:03.135 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.135 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.135 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.135 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.135 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392902+0000", 2024-04-26T01:08:03.135 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392903+0000", 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9400000000000005e-07 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392906+0000", 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6060000000000001e-06 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392914+0000", 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.9410000000000007e-06 2024-04-26T01:08:03.136 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.137 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.137 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.137 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.392919+0000", 2024-04-26T01:08:03.137 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4879999999999998e-06 2024-04-26T01:08:03.137 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.137 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.137 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.137 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.395847+0000", 2024-04-26T01:08:03.137 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0029283260000000002 2024-04-26T01:08:03.137 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.137 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.137 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:03.137 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.395944+0000", 2024-04-26T01:08:03.138 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6405999999999998e-05 2024-04-26T01:08:03.138 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.138 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.138 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.138 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.399500+0000", 2024-04-26T01:08:03.138 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035560230000000002 2024-04-26T01:08:03.138 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.138 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.138 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:03.138 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.399624+0000", 2024-04-26T01:08:03.138 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012399899999999999 2024-04-26T01:08:03.138 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.138 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.138 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.139 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.402992+0000", 2024-04-26T01:08:03.139 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033682159999999998 2024-04-26T01:08:03.139 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.139 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.139 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.139 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.139 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.139 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs1 33/30 ECSubWrite(tid=21, reqid=client.4521.0:89, at_version=33'8, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:08:03.139 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.402003+0000", 2024-04-26T01:08:03.139 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.6663288500000002, 2024-04-26T01:08:03.139 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033864810000000002, 2024-04-26T01:08:03.139 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.139 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.140 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.140 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.140 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.140 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.140 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.402003+0000", 2024-04-26T01:08:03.140 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.140 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.140 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.140 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.140 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.402003+0000", 2024-04-26T01:08:03.140 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.140 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.141 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.141 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.141 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.402004+0000", 2024-04-26T01:08:03.141 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8699999999999995e-07 2024-04-26T01:08:03.141 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.141 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.141 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.141 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.402018+0000", 2024-04-26T01:08:03.141 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4749999999999999e-05 2024-04-26T01:08:03.141 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.141 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.141 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.142 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.404174+0000", 2024-04-26T01:08:03.142 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0021553290000000001 2024-04-26T01:08:03.142 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.142 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.142 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.142 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.404177+0000", 2024-04-26T01:08:03.142 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3220000000000001e-06 2024-04-26T01:08:03.142 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.142 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.142 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.142 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.404661+0000", 2024-04-26T01:08:03.142 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00048436699999999998 2024-04-26T01:08:03.143 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.143 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.143 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.143 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.404674+0000", 2024-04-26T01:08:03.143 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2644e-05 2024-04-26T01:08:03.143 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.143 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.143 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.143 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405358+0000", 2024-04-26T01:08:03.143 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00068389600000000003 2024-04-26T01:08:03.143 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.143 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.144 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.144 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405390+0000", 2024-04-26T01:08:03.144 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1686000000000001e-05 2024-04-26T01:08:03.144 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.144 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.144 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.144 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.144 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.144 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:94 3.4s0 3:211cd322:::smithi16428812-32:head [append 0~524288,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:03.144 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.404969+0000", 2024-04-26T01:08:03.144 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.6633633140000001, 2024-04-26T01:08:03.144 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.005253738, 2024-04-26T01:08:03.144 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.144 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.145 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:03.145 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:03.145 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:03.145 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:03.145 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 94 2024-04-26T01:08:03.145 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.145 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.145 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.145 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.145 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.404969+0000", 2024-04-26T01:08:03.145 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.145 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.145 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.146 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.146 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.404969+0000", 2024-04-26T01:08:03.146 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.146 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.146 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.146 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.146 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.404970+0000", 2024-04-26T01:08:03.146 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.615e-06 2024-04-26T01:08:03.146 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.146 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.146 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.146 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405492+0000", 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00052180599999999998 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405501+0000", 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7290000000000006e-06 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405506+0000", 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1390000000000003e-06 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405543+0000", 2024-04-26T01:08:03.147 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7321999999999999e-05 2024-04-26T01:08:03.148 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.148 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.148 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:03.148 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405687+0000", 2024-04-26T01:08:03.148 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014419899999999999 2024-04-26T01:08:03.148 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.148 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.148 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.148 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.406527+0000", 2024-04-26T01:08:03.148 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000839757 2024-04-26T01:08:03.148 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.148 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.148 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.148 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.407826+0000", 2024-04-26T01:08:03.149 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0012983809999999999 2024-04-26T01:08:03.149 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.149 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.149 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:03.149 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.410155+0000", 2024-04-26T01:08:03.149 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0023298450000000001 2024-04-26T01:08:03.149 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.149 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.149 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.149 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.410222+0000", 2024-04-26T01:08:03.149 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6945000000000004e-05 2024-04-26T01:08:03.149 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.149 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.149 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.149 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.150 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.150 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:95 3.4s0 3:211cd322:::smithi16428812-32:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:03.150 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.405518+0000", 2024-04-26T01:08:03.150 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.6628137650000001, 2024-04-26T01:08:03.150 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0051108880000000001, 2024-04-26T01:08:03.150 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.150 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.150 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:03.150 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:03.150 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:03.150 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:03.150 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 95 2024-04-26T01:08:03.150 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.150 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.151 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.151 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.151 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405518+0000", 2024-04-26T01:08:03.151 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.151 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.151 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.151 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.151 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405518+0000", 2024-04-26T01:08:03.151 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.151 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.151 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.151 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.151 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405519+0000", 2024-04-26T01:08:03.151 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0100000000000004e-07 2024-04-26T01:08:03.151 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405522+0000", 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.303e-06 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405529+0000", 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9009999999999997e-06 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405535+0000", 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0070000000000003e-06 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.152 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.153 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.153 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.407157+0000", 2024-04-26T01:08:03.153 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0016213709999999999 2024-04-26T01:08:03.153 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.153 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.153 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:03.153 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.407209+0000", 2024-04-26T01:08:03.153 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2015000000000003e-05 2024-04-26T01:08:03.153 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.153 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.153 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.153 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.407488+0000", 2024-04-26T01:08:03.153 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000279475 2024-04-26T01:08:03.153 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.153 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.154 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.154 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.408295+0000", 2024-04-26T01:08:03.154 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000807317 2024-04-26T01:08:03.154 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.154 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.154 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:03.154 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.410556+0000", 2024-04-26T01:08:03.154 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022604280000000001 2024-04-26T01:08:03.154 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.154 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.154 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.154 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.410629+0000", 2024-04-26T01:08:03.154 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3269999999999995e-05 2024-04-26T01:08:03.154 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.154 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:96 3.4s0 3:211cd322:::smithi16428812-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.405544+0000", 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.6627874650000001, 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0076542160000000001, 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 96 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.155 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405544+0000", 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405544+0000", 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405545+0000", 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8299999999999996e-07 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.156 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.157 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.157 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405549+0000", 2024-04-26T01:08:03.157 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8879999999999999e-06 2024-04-26T01:08:03.157 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.157 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.157 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.157 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405556+0000", 2024-04-26T01:08:03.157 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2840000000000001e-06 2024-04-26T01:08:03.157 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.157 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.157 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.157 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.405561+0000", 2024-04-26T01:08:03.157 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2930000000000002e-06 2024-04-26T01:08:03.157 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.157 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.158 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.158 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.407743+0000", 2024-04-26T01:08:03.158 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002182692 2024-04-26T01:08:03.158 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.158 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.158 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:03.158 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.407809+0000", 2024-04-26T01:08:03.158 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5456999999999993e-05 2024-04-26T01:08:03.158 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.158 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.158 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.158 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411153+0000", 2024-04-26T01:08:03.158 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033444870000000002 2024-04-26T01:08:03.158 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.159 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.159 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:03.159 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411239+0000", 2024-04-26T01:08:03.159 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5235999999999995e-05 2024-04-26T01:08:03.159 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.159 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.159 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.159 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413199+0000", 2024-04-26T01:08:03.159 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0019600960000000001 2024-04-26T01:08:03.159 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.159 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.159 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.159 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.159 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.160 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:163 3.4s0 3:2d34bde3:::smithi16428812-29:head [create,setxattr eWpanQhTTi^PdqZriWXO\\ppRdZhpliZa`ZaisXa[ (208) in=40b,setxattr n\\Rcik[oWUoSYooSn`\\pPbdNcZsjcZ[deNcYkmlj (1998) in=40b,setxattr anlcchUp`RkaTlVWpf^o^`PsaRZaott[QbYT_Q^q (1191) in=40b,setxattr canQSZSejofUVgorbfikse\\a`qTe\\bcZajjt[a_a (383) in=40b,setxattr skRVbtnpqp[s\\t_lQ`do`XZobkkm`PNWkcjUn[i] (1577) in=40b,setxattr aekXNRb\\nfkiscaaScp^eciOpid_lTXlaTYcnicX (70) in=40b,setxattr `OcOc[dUgmaciYrfadObahbn`NQhsd`hf\\qOhXbY (1718) in=40b,setxattr deoPasjsZXVrpnUVYbdWto`j_]QtarctfcPNSrhT (1611) in=40b,setxattr c[qN]WVrjWj^SWbZne (1157) in=18b,setxattr OOSfdd_`ntjtlSe[_aWf\\ndnqWgY^`N`QhZh^lpg (10804) in=40b,setxattr bd`aeUsmZslNtYa]pfk]^_WhWN\\ftS^X[rk\\qjZg (771) in=40b,setxattr cTNeeaR^rVPPokNtfjObY`jqRmqhON^\\oPsXe]pk (945) in=40b,setxattr epk_SsTm`NeYqkYW]bsaN[rcrbYhPcqiSYga^\\da (314) in=40b,setxattr UTUd\\Zs]Nin_NRopmbUcSRUi[gnRr]NQkWjr[Sq_ (1717) in=40b,setxattr ao[qXkmgXYf^ZQai]bPpeQhdNRtgoQlmssaWjTPa (1331) in=40b,setxattr clZY[aNUaSoXatU^SRVdQ]aPQae^iS_pUaorl^UY (174) in=40b,setxattr lfXUaadNlje^`SaRY]T]\\RYbWdV[iWffn^mWhhjb (175) in=40b,setxattr [Wh_egdbRmeaX^e_SoZ]hNacaUfa`TV`^cgega_` (1366) in=40b,setxattr hTmfa\\lj`gUWdSY^m\\_hZkRUfiZPeanZVWVmoi]V (1157) in=40b,setxattr meTbcir_og\\bsWZ`[gU^cpSj_ZjnTVUf\\qk^VTcj (420) in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:03.160 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.495526+0000", 2024-04-26T01:08:03.160 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.5728056239999999, 2024-04-26T01:08:03.160 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0049345680000000003, 2024-04-26T01:08:03.160 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.160 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.160 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:03.160 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:03.160 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:03.160 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:03.160 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 163 2024-04-26T01:08:03.160 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.160 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.160 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.160 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.161 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.495526+0000", 2024-04-26T01:08:03.161 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.161 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.161 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.161 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.161 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.495526+0000", 2024-04-26T01:08:03.161 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.161 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.161 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.161 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.161 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.495528+0000", 2024-04-26T01:08:03.161 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9929999999999998e-06 2024-04-26T01:08:03.161 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.161 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.161 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.162 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.495562+0000", 2024-04-26T01:08:03.162 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3256000000000001e-05 2024-04-26T01:08:03.162 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.162 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.162 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.162 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.495579+0000", 2024-04-26T01:08:03.162 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7532000000000001e-05 2024-04-26T01:08:03.162 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.162 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.162 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.162 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.495587+0000", 2024-04-26T01:08:03.162 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7989999999999994e-06 2024-04-26T01:08:03.162 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.162 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.162 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.163 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.495649+0000", 2024-04-26T01:08:03.163 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1682999999999999e-05 2024-04-26T01:08:03.163 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.163 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.163 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:03.163 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.495769+0000", 2024-04-26T01:08:03.163 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000120217 2024-04-26T01:08:03.163 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.163 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.163 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.163 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.496667+0000", 2024-04-26T01:08:03.163 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00089777599999999998 2024-04-26T01:08:03.163 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.163 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.163 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.164 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.498214+0000", 2024-04-26T01:08:03.164 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015474309999999999 2024-04-26T01:08:03.164 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.164 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.164 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:03.164 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.500339+0000", 2024-04-26T01:08:03.164 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0021250230000000002 2024-04-26T01:08:03.164 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.164 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.164 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.164 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.500461+0000", 2024-04-26T01:08:03.164 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000121858 2024-04-26T01:08:03.164 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.164 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.164 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.5s1 33/30 ECSubWrite(tid=10, reqid=client.4521.0:180, at_version=33'7, trim_to=0'0, roll_forward_to=33'5))", 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.531511+0000", 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.5368208450000003, 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003650361, 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.531511+0000", 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.165 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.166 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.166 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.531511+0000", 2024-04-26T01:08:03.166 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.166 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.166 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.166 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.166 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.531512+0000", 2024-04-26T01:08:03.166 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3090000000000001e-06 2024-04-26T01:08:03.166 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.166 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.166 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.166 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.532362+0000", 2024-04-26T01:08:03.166 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00084943099999999999 2024-04-26T01:08:03.166 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.167 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.167 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.167 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.532409+0000", 2024-04-26T01:08:03.167 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7058999999999998e-05 2024-04-26T01:08:03.167 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.167 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.167 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.167 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.532455+0000", 2024-04-26T01:08:03.167 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6066000000000002e-05 2024-04-26T01:08:03.167 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.167 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.167 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.167 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.532578+0000", 2024-04-26T01:08:03.167 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000123418 2024-04-26T01:08:03.167 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.532588+0000", 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.5840000000000001e-06 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.535120+0000", 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002531958 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.535161+0000", 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1535999999999997e-05 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.168 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.169 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.169 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.169 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.169 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:187 3.4s0 3:211cd322:::smithi16428812-32:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:03.169 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.532341+0000", 2024-04-26T01:08:03.169 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.5359907379999997, 2024-04-26T01:08:03.169 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0055541690000000003, 2024-04-26T01:08:03.169 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.169 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.169 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:03.169 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:03.169 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:03.169 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:03.169 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 187 2024-04-26T01:08:03.170 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.170 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.170 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.170 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.170 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.532341+0000", 2024-04-26T01:08:03.170 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.170 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.170 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.170 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.170 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.532341+0000", 2024-04-26T01:08:03.170 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.170 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.170 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.170 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.171 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.532344+0000", 2024-04-26T01:08:03.171 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7209999999999999e-06 2024-04-26T01:08:03.171 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.171 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.171 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.171 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.532351+0000", 2024-04-26T01:08:03.171 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9659999999999999e-06 2024-04-26T01:08:03.171 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.171 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.171 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.171 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.532379+0000", 2024-04-26T01:08:03.171 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8036999999999999e-05 2024-04-26T01:08:03.171 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.172 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.172 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.172 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.532389+0000", 2024-04-26T01:08:03.172 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6719999999999999e-06 2024-04-26T01:08:03.172 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.172 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.172 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.172 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.532455+0000", 2024-04-26T01:08:03.172 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6744999999999999e-05 2024-04-26T01:08:03.172 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.172 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.172 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:03.172 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.532713+0000", 2024-04-26T01:08:03.172 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000258146 2024-04-26T01:08:03.172 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.173 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.173 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.173 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.533218+0000", 2024-04-26T01:08:03.173 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00050461799999999997 2024-04-26T01:08:03.173 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.173 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.173 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.173 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.535106+0000", 2024-04-26T01:08:03.173 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0018882199999999999 2024-04-26T01:08:03.173 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.173 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.173 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:03.173 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.537811+0000", 2024-04-26T01:08:03.173 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027049029999999998 2024-04-26T01:08:03.173 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.174 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.174 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.174 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.537895+0000", 2024-04-26T01:08:03.174 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4141000000000006e-05 2024-04-26T01:08:03.174 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.174 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.174 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.174 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.174 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.174 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:03.175 INFO:tasks.rados.rados.0.smithi164.stdout:88: finishing copy_from to smithi16428812-8 2024-04-26T01:08:03.175 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 8 v 4 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-26T01:08:03.176 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:08:03.177 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:03.173+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:03.276 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:03.276+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:03.336 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-04-26T01:08:03.342 INFO:tasks.rados.rados.0.smithi164.stdout:86: done (3 left) 2024-04-26T01:08:03.342 INFO:tasks.rados.rados.0.smithi164.stdout:87: done (2 left) 2024-04-26T01:08:03.342 INFO:tasks.rados.rados.0.smithi164.stdout:88: done (1 left) 2024-04-26T01:08:03.342 INFO:tasks.rados.rados.0.smithi164.stdout:89: done (0 left) 2024-04-26T01:08:03.342 INFO:tasks.rados.rados.0.smithi164.stdout:90: read oid 42 snap -1 2024-04-26T01:08:03.342 INFO:tasks.rados.rados.0.smithi164.stdout:90: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-26T01:08:03.342 INFO:tasks.rados.rados.0.smithi164.stdout:91: read oid 24 snap -1 2024-04-26T01:08:03.342 INFO:tasks.rados.rados.0.smithi164.stdout:91: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-26T01:08:03.343 INFO:tasks.rados.rados.0.smithi164.stdout:92: delete oid 11 current snap is 3 2024-04-26T01:08:03.352 INFO:tasks.rados.rados.0.smithi164.stdout:90: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-26T01:08:03.358 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:03.358 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:03.358 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-04-26T01:08:03.358 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:03.373 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:08:03.374 INFO:tasks.rados.rados.0.smithi164.stdout:91: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-26T01:08:03.394 INFO:tasks.rados.rados.0.smithi164.stdout:90: done (2 left) 2024-04-26T01:08:03.394 INFO:tasks.rados.rados.0.smithi164.stdout:91: done (1 left) 2024-04-26T01:08:03.395 INFO:tasks.rados.rados.0.smithi164.stdout:92: done (0 left) 2024-04-26T01:08:03.395 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 46 current snap is 3 2024-04-26T01:08:03.395 INFO:tasks.rados.rados.0.smithi164.stdout:93: seq_num 64 ranges {770048=704512,1474560=376832} 2024-04-26T01:08:03.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:03.417+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:03.418 INFO:tasks.rados.rados.0.smithi164.stdout:93: writing smithi16428812-46 from 770048 to 1474560 tid 1 2024-04-26T01:08:03.426 INFO:tasks.rados.rados.0.smithi164.stdout:93: writing smithi16428812-46 from 1474560 to 1851392 tid 2 2024-04-26T01:08:03.426 INFO:tasks.rados.rados.0.smithi164.stdout:94: copy_from oid 45 from oid 19 current snap is 3 2024-04-26T01:08:03.426 INFO:tasks.rados.rados.0.smithi164.stdout:95: rollback oid 10 current snap is 3 2024-04-26T01:08:03.426 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 10 to 0 2024-04-26T01:08:03.426 INFO:tasks.rados.rados.0.smithi164.stdout:96: read oid 22 snap -1 2024-04-26T01:08:03.426 INFO:tasks.rados.rados.0.smithi164.stdout:96: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-26T01:08:03.426 INFO:tasks.rados.rados.0.smithi164.stdout:97: rollback oid 8 current snap is 3 2024-04-26T01:08:03.426 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 8 to 0 2024-04-26T01:08:03.426 INFO:tasks.rados.rados.0.smithi164.stdout:98: setattr oid 34 current snap is 3 2024-04-26T01:08:03.432 INFO:tasks.rados.rados.0.smithi164.stdout:99: snap_remove snap 2 2024-04-26T01:08:03.479 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:03.477+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:03.543 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:03.543 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:03.543 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-04-26T01:08:03.543 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-04-26T01:08:03.543 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:03.557 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:08:03.580 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:03.580+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:03.680 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:03.677+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:03.732 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:03.733 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:03.733 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:03.733 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:03.733 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.733 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 33/30 ECSubWrite(tid=0, reqid=client.4521.0:22, at_version=33'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:03.733 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.310594+0000", 2024-04-26T01:08:03.733 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.4183179560000001, 2024-04-26T01:08:03.733 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035400710000000001, 2024-04-26T01:08:03.733 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.733 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.733 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.733 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.733 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.734 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.734 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310594+0000", 2024-04-26T01:08:03.734 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.734 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.734 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.734 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.734 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310594+0000", 2024-04-26T01:08:03.734 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.734 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.734 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.734 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.734 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310594+0000", 2024-04-26T01:08:03.734 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7500000000000001e-07 2024-04-26T01:08:03.734 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.735 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.735 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.735 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310673+0000", 2024-04-26T01:08:03.735 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8784999999999999e-05 2024-04-26T01:08:03.735 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.735 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.735 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.735 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310747+0000", 2024-04-26T01:08:03.735 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3899999999999994e-05 2024-04-26T01:08:03.735 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.735 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.735 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.735 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310763+0000", 2024-04-26T01:08:03.735 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5769000000000001e-05 2024-04-26T01:08:03.735 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.736 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.736 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.736 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310858+0000", 2024-04-26T01:08:03.736 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.5383000000000005e-05 2024-04-26T01:08:03.736 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.736 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.736 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.736 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310871+0000", 2024-04-26T01:08:03.736 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2887e-05 2024-04-26T01:08:03.736 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.736 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.736 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.736 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314085+0000", 2024-04-26T01:08:03.736 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032136249999999999 2024-04-26T01:08:03.736 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.737 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.737 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.737 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314134+0000", 2024-04-26T01:08:03.737 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9447000000000003e-05 2024-04-26T01:08:03.737 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.737 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.737 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.737 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.737 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.737 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 33/30 ECSubWrite(tid=1, reqid=client.4521.0:23, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:03.737 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.311455+0000", 2024-04-26T01:08:03.737 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.417457239, 2024-04-26T01:08:03.737 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027249739999999998, 2024-04-26T01:08:03.737 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.737 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.738 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.738 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.738 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.738 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.738 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311455+0000", 2024-04-26T01:08:03.738 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.738 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.738 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.738 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.738 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311455+0000", 2024-04-26T01:08:03.738 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.738 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.738 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.738 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.739 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311456+0000", 2024-04-26T01:08:03.739 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.012e-06 2024-04-26T01:08:03.739 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.739 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.739 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.739 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311479+0000", 2024-04-26T01:08:03.739 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3456e-05 2024-04-26T01:08:03.739 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.740 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.740 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.740 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311527+0000", 2024-04-26T01:08:03.740 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7827999999999997e-05 2024-04-26T01:08:03.740 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.740 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.740 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.740 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311540+0000", 2024-04-26T01:08:03.740 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2921e-05 2024-04-26T01:08:03.740 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.740 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.740 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.740 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311641+0000", 2024-04-26T01:08:03.741 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000101034 2024-04-26T01:08:03.741 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.741 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.741 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.741 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311654+0000", 2024-04-26T01:08:03.741 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2781e-05 2024-04-26T01:08:03.741 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.741 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.741 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.741 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314144+0000", 2024-04-26T01:08:03.741 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024907200000000001 2024-04-26T01:08:03.741 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.741 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.741 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.741 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314180+0000", 2024-04-26T01:08:03.742 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.5222000000000003e-05 2024-04-26T01:08:03.742 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.742 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.742 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.742 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.742 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.742 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 32/30 ECSubWrite(tid=1, reqid=client.4521.0:25, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:03.742 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.312642+0000", 2024-04-26T01:08:03.742 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.4162697279999996, 2024-04-26T01:08:03.742 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032273459999999999, 2024-04-26T01:08:03.743 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.743 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.743 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.743 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.743 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.743 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.743 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312642+0000", 2024-04-26T01:08:03.743 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.743 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.743 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.743 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.744 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312642+0000", 2024-04-26T01:08:03.744 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.744 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.744 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.744 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.744 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312644+0000", 2024-04-26T01:08:03.744 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2720000000000001e-06 2024-04-26T01:08:03.744 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.744 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.744 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.744 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312878+0000", 2024-04-26T01:08:03.744 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00023409 2024-04-26T01:08:03.744 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.744 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.744 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.745 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312930+0000", 2024-04-26T01:08:03.745 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2799000000000001e-05 2024-04-26T01:08:03.745 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.745 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.745 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.745 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312944+0000", 2024-04-26T01:08:03.745 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3687e-05 2024-04-26T01:08:03.745 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.745 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.745 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.745 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.313045+0000", 2024-04-26T01:08:03.745 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000100782 2024-04-26T01:08:03.745 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.745 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.745 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.746 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.313058+0000", 2024-04-26T01:08:03.746 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2785999999999999e-05 2024-04-26T01:08:03.746 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.746 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.746 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.746 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.315782+0000", 2024-04-26T01:08:03.746 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027242439999999998 2024-04-26T01:08:03.746 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.746 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.746 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.746 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.315870+0000", 2024-04-26T01:08:03.746 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7686e-05 2024-04-26T01:08:03.746 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.746 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.746 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.747 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.747 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.747 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 33/30 ECSubWrite(tid=4, reqid=client.4521.0:26, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:03.747 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.326410+0000", 2024-04-26T01:08:03.747 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.4025023130000003, 2024-04-26T01:08:03.747 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0013993580000000001, 2024-04-26T01:08:03.747 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.747 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.747 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.747 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.747 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.747 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.748 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.326410+0000", 2024-04-26T01:08:03.748 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.748 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.748 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.748 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.748 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.326410+0000", 2024-04-26T01:08:03.748 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.748 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.748 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.748 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.748 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.326411+0000", 2024-04-26T01:08:03.748 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3999999999999999e-06 2024-04-26T01:08:03.749 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.749 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.749 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.749 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.326429+0000", 2024-04-26T01:08:03.749 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7665000000000001e-05 2024-04-26T01:08:03.749 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.749 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.749 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.749 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.326486+0000", 2024-04-26T01:08:03.749 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7695999999999998e-05 2024-04-26T01:08:03.749 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.749 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.749 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.749 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.326515+0000", 2024-04-26T01:08:03.749 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8206000000000001e-05 2024-04-26T01:08:03.750 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.750 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.750 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.750 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.326609+0000", 2024-04-26T01:08:03.750 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4035000000000003e-05 2024-04-26T01:08:03.750 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.750 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.750 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.750 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.326623+0000", 2024-04-26T01:08:03.750 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4075e-05 2024-04-26T01:08:03.750 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.751 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.751 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.751 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.327755+0000", 2024-04-26T01:08:03.751 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001132147 2024-04-26T01:08:03.751 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.751 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.752 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.752 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.327809+0000", 2024-04-26T01:08:03.752 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4134e-05 2024-04-26T01:08:03.752 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.752 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.752 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.752 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.752 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.753 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.6s4 33/30 ECSubWrite(tid=0, reqid=client.4521.0:43, at_version=33'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:03.753 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.348025+0000", 2024-04-26T01:08:03.753 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.3808871700000003, 2024-04-26T01:08:03.753 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001564726, 2024-04-26T01:08:03.753 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.753 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.753 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.753 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.753 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.753 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.753 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.348025+0000", 2024-04-26T01:08:03.753 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.753 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.753 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.754 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.754 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.348025+0000", 2024-04-26T01:08:03.754 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.754 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.754 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.754 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.754 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.348025+0000", 2024-04-26T01:08:03.754 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0400000000000002e-07 2024-04-26T01:08:03.754 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.754 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.754 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.754 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.348089+0000", 2024-04-26T01:08:03.754 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4079999999999996e-05 2024-04-26T01:08:03.754 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.754 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.755 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.755 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.348153+0000", 2024-04-26T01:08:03.755 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3628999999999997e-05 2024-04-26T01:08:03.755 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.755 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.755 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.755 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.348166+0000", 2024-04-26T01:08:03.755 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3529999999999999e-05 2024-04-26T01:08:03.755 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.755 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.755 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.755 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.348306+0000", 2024-04-26T01:08:03.755 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013938700000000001 2024-04-26T01:08:03.755 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.755 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.756 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.756 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.348320+0000", 2024-04-26T01:08:03.756 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4021e-05 2024-04-26T01:08:03.756 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.756 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.756 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.756 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349563+0000", 2024-04-26T01:08:03.756 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001243678 2024-04-26T01:08:03.756 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.756 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.756 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.756 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349590+0000", 2024-04-26T01:08:03.756 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6097000000000001e-05 2024-04-26T01:08:03.756 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.756 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.757 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.757 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.757 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.757 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.6s4 33/30 ECSubWrite(tid=1, reqid=client.4521.0:44, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:03.757 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.349184+0000", 2024-04-26T01:08:03.758 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.3797275110000005, 2024-04-26T01:08:03.758 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001341538, 2024-04-26T01:08:03.758 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.758 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.758 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.758 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.758 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.758 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.758 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349184+0000", 2024-04-26T01:08:03.758 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.758 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.758 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.758 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.758 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349184+0000", 2024-04-26T01:08:03.759 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.759 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.759 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.759 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.759 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349186+0000", 2024-04-26T01:08:03.759 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0920000000000001e-06 2024-04-26T01:08:03.759 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.759 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.759 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.759 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349207+0000", 2024-04-26T01:08:03.759 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1430999999999999e-05 2024-04-26T01:08:03.759 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.759 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.759 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.759 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349257+0000", 2024-04-26T01:08:03.760 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.0225000000000001e-05 2024-04-26T01:08:03.760 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.760 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.760 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.760 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349268+0000", 2024-04-26T01:08:03.760 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1044e-05 2024-04-26T01:08:03.760 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.760 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.760 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.760 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349346+0000", 2024-04-26T01:08:03.760 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7917999999999995e-05 2024-04-26T01:08:03.760 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.760 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.760 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.760 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349359+0000", 2024-04-26T01:08:03.761 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2522e-05 2024-04-26T01:08:03.761 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.761 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.761 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.761 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.350485+0000", 2024-04-26T01:08:03.761 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0011258869999999999 2024-04-26T01:08:03.761 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.761 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.762 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.762 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.350526+0000", 2024-04-26T01:08:03.762 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1418999999999998e-05 2024-04-26T01:08:03.762 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.762 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.762 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.762 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.762 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.762 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 33/30 ECSubWrite(tid=4, reqid=client.4521.0:61, at_version=33'3, trim_to=0'0, roll_forward_to=33'2))", 2024-04-26T01:08:03.762 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.361802+0000", 2024-04-26T01:08:03.762 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.3671097220000004, 2024-04-26T01:08:03.762 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0016449189999999999, 2024-04-26T01:08:03.762 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.762 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.762 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.763 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.763 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.763 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.763 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.361802+0000", 2024-04-26T01:08:03.763 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.763 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.763 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.763 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.763 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.361802+0000", 2024-04-26T01:08:03.763 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.763 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.763 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.763 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.763 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.361804+0000", 2024-04-26T01:08:03.764 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2589999999999999e-06 2024-04-26T01:08:03.764 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.764 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.764 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.764 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.361972+0000", 2024-04-26T01:08:03.764 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016893999999999999 2024-04-26T01:08:03.764 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.764 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.764 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.764 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.362044+0000", 2024-04-26T01:08:03.764 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1662000000000006e-05 2024-04-26T01:08:03.764 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.764 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.764 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.765 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.362059+0000", 2024-04-26T01:08:03.765 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4943000000000001e-05 2024-04-26T01:08:03.765 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.765 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.765 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.765 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.362144+0000", 2024-04-26T01:08:03.765 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4512000000000007e-05 2024-04-26T01:08:03.766 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.766 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.766 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.766 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.362157+0000", 2024-04-26T01:08:03.766 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3039e-05 2024-04-26T01:08:03.766 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.766 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.766 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.766 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.363384+0000", 2024-04-26T01:08:03.766 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001227141 2024-04-26T01:08:03.767 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.767 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.767 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.767 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.363447+0000", 2024-04-26T01:08:03.767 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3423000000000006e-05 2024-04-26T01:08:03.767 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.767 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.767 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.767 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.767 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.767 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 33/30 ECSubWrite(tid=14, reqid=client.4521.0:67, at_version=33'5, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:08:03.767 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.369300+0000", 2024-04-26T01:08:03.768 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.3596116990000002, 2024-04-26T01:08:03.768 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0021196639999999998, 2024-04-26T01:08:03.768 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.768 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.768 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.768 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.768 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.768 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.768 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369300+0000", 2024-04-26T01:08:03.768 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.768 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.768 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.768 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.769 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369300+0000", 2024-04-26T01:08:03.769 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.769 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.769 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.769 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.769 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369302+0000", 2024-04-26T01:08:03.769 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6780000000000001e-06 2024-04-26T01:08:03.769 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.769 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.769 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.769 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369545+0000", 2024-04-26T01:08:03.769 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00024281699999999999 2024-04-26T01:08:03.769 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.770 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.770 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.770 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369586+0000", 2024-04-26T01:08:03.770 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1696000000000002e-05 2024-04-26T01:08:03.770 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.770 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.770 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.770 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369598+0000", 2024-04-26T01:08:03.770 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2007999999999999e-05 2024-04-26T01:08:03.770 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.770 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.770 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.770 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369681+0000", 2024-04-26T01:08:03.771 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2170999999999996e-05 2024-04-26T01:08:03.771 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.771 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.771 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.771 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369696+0000", 2024-04-26T01:08:03.771 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5224e-05 2024-04-26T01:08:03.771 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.771 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.771 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.771 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.371177+0000", 2024-04-26T01:08:03.771 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0014809720000000001 2024-04-26T01:08:03.771 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.771 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.771 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.772 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.371420+0000", 2024-04-26T01:08:03.772 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000243098 2024-04-26T01:08:03.772 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.772 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.772 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.772 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.772 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.772 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.6s4 33/30 ECSubWrite(tid=4, reqid=client.4521.0:70, at_version=33'3, trim_to=0'0, roll_forward_to=33'2))", 2024-04-26T01:08:03.772 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.373375+0000", 2024-04-26T01:08:03.772 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.355537215, 2024-04-26T01:08:03.772 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002076091, 2024-04-26T01:08:03.772 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.772 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.772 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.773 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.773 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.773 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.773 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.373375+0000", 2024-04-26T01:08:03.773 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.773 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.773 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.773 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.773 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.373375+0000", 2024-04-26T01:08:03.773 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.773 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.773 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.773 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.773 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.373376+0000", 2024-04-26T01:08:03.774 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.108e-06 2024-04-26T01:08:03.774 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.774 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.774 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.774 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.373513+0000", 2024-04-26T01:08:03.774 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013695999999999999 2024-04-26T01:08:03.774 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.774 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.774 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.774 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.373564+0000", 2024-04-26T01:08:03.774 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.0861e-05 2024-04-26T01:08:03.774 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.774 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.774 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.774 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.373576+0000", 2024-04-26T01:08:03.775 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1902999999999999e-05 2024-04-26T01:08:03.775 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.775 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.775 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.775 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.373662+0000", 2024-04-26T01:08:03.775 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6556000000000003e-05 2024-04-26T01:08:03.775 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.775 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.775 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.775 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.373674+0000", 2024-04-26T01:08:03.775 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.147e-05 2024-04-26T01:08:03.775 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.775 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.775 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.775 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.375410+0000", 2024-04-26T01:08:03.776 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0017359330000000001 2024-04-26T01:08:03.776 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.776 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.776 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.776 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.375451+0000", 2024-04-26T01:08:03.776 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1300000000000001e-05 2024-04-26T01:08:03.776 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.776 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.776 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.776 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.776 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.776 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 33/30 ECSubWrite(tid=20, reqid=client.4521.0:88, at_version=33'7, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:08:03.776 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.398260+0000", 2024-04-26T01:08:03.776 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.3306516080000002, 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0020125469999999999, 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398260+0000", 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398260+0000", 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.777 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.778 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.778 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398262+0000", 2024-04-26T01:08:03.778 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.562e-06 2024-04-26T01:08:03.778 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.778 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.778 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.778 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398462+0000", 2024-04-26T01:08:03.778 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019968500000000001 2024-04-26T01:08:03.778 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.778 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.778 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.778 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398515+0000", 2024-04-26T01:08:03.778 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3724999999999998e-05 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398534+0000", 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8362000000000001e-05 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398612+0000", 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7953000000000004e-05 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398627+0000", 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5454000000000001e-05 2024-04-26T01:08:03.779 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.780 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.780 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.780 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.400232+0000", 2024-04-26T01:08:03.780 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001604474 2024-04-26T01:08:03.780 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.780 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.780 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.780 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.400273+0000", 2024-04-26T01:08:03.780 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1331999999999997e-05 2024-04-26T01:08:03.780 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.780 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.780 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.780 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.780 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.780 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 33/30 ECSubWrite(tid=22, reqid=client.4521.0:97, at_version=33'9, trim_to=0'0, roll_forward_to=33'7))", 2024-04-26T01:08:03.781 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.411236+0000", 2024-04-26T01:08:03.781 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.3176762259999997, 2024-04-26T01:08:03.781 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001698829, 2024-04-26T01:08:03.781 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.781 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.781 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.781 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.781 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.781 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.781 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411236+0000", 2024-04-26T01:08:03.781 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.781 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.781 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.781 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.782 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411236+0000", 2024-04-26T01:08:03.782 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.782 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.782 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.782 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.782 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411237+0000", 2024-04-26T01:08:03.782 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.658e-06 2024-04-26T01:08:03.782 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.782 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.782 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.782 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411646+0000", 2024-04-26T01:08:03.782 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00040855299999999998 2024-04-26T01:08:03.782 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.782 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.782 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.783 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411694+0000", 2024-04-26T01:08:03.783 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8486999999999998e-05 2024-04-26T01:08:03.783 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.783 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.783 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.783 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411704+0000", 2024-04-26T01:08:03.783 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0010000000000001e-05 2024-04-26T01:08:03.783 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.783 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.783 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.783 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411835+0000", 2024-04-26T01:08:03.783 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013073199999999999 2024-04-26T01:08:03.783 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.784 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.784 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.784 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411846+0000", 2024-04-26T01:08:03.784 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0577999999999999e-05 2024-04-26T01:08:03.784 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.784 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.784 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.784 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.412887+0000", 2024-04-26T01:08:03.784 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001041134 2024-04-26T01:08:03.784 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.784 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.784 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.784 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.412935+0000", 2024-04-26T01:08:03.784 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7676999999999998e-05 2024-04-26T01:08:03.784 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.785 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.785 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.785 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.785 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.785 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.6s4 33/30 ECSubWrite(tid=8, reqid=client.4521.0:112, at_version=33'5, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:08:03.785 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.433344+0000", 2024-04-26T01:08:03.785 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.2955676519999999, 2024-04-26T01:08:03.785 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0019042270000000001, 2024-04-26T01:08:03.785 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.785 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.785 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.785 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.785 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.785 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.786 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.433344+0000", 2024-04-26T01:08:03.786 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.786 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.786 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.786 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.786 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.433344+0000", 2024-04-26T01:08:03.786 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.786 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.786 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.786 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.786 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.433346+0000", 2024-04-26T01:08:03.786 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.358e-06 2024-04-26T01:08:03.786 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.786 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.787 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.787 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.433496+0000", 2024-04-26T01:08:03.787 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015016300000000001 2024-04-26T01:08:03.787 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.787 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.787 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.787 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.433549+0000", 2024-04-26T01:08:03.787 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3553e-05 2024-04-26T01:08:03.787 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.787 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.787 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.787 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.433561+0000", 2024-04-26T01:08:03.787 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2064e-05 2024-04-26T01:08:03.787 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.788 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.788 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.788 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.433653+0000", 2024-04-26T01:08:03.788 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2033000000000005e-05 2024-04-26T01:08:03.788 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.788 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.788 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.788 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.433664+0000", 2024-04-26T01:08:03.788 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0978000000000001e-05 2024-04-26T01:08:03.788 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.788 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.788 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.788 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.435213+0000", 2024-04-26T01:08:03.788 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015485410000000001 2024-04-26T01:08:03.789 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.789 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.789 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.789 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.435249+0000", 2024-04-26T01:08:03.789 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.5537000000000002e-05 2024-04-26T01:08:03.789 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.789 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.789 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.789 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.789 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.789 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs3 33/30 ECSubWrite(tid=31, reqid=client.4521.0:118, at_version=33'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:03.789 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.444809+0000", 2024-04-26T01:08:03.789 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.2841032749999997, 2024-04-26T01:08:03.789 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0013569400000000001, 2024-04-26T01:08:03.789 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.790 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.790 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.790 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.790 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.790 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.790 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444809+0000", 2024-04-26T01:08:03.790 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.790 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.790 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.790 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.790 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444809+0000", 2024-04-26T01:08:03.790 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.790 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.790 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.791 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.791 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444809+0000", 2024-04-26T01:08:03.791 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3799999999999999e-07 2024-04-26T01:08:03.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.791 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.791 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.791 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444951+0000", 2024-04-26T01:08:03.791 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014165099999999999 2024-04-26T01:08:03.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.791 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.791 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.791 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.445004+0000", 2024-04-26T01:08:03.791 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3633e-05 2024-04-26T01:08:03.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.792 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.792 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.792 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.445015+0000", 2024-04-26T01:08:03.792 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1211e-05 2024-04-26T01:08:03.792 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.792 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.792 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.792 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.445097+0000", 2024-04-26T01:08:03.792 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1811999999999995e-05 2024-04-26T01:08:03.792 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.792 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.792 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.792 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.445108+0000", 2024-04-26T01:08:03.792 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1069e-05 2024-04-26T01:08:03.792 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.793 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.793 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.793 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446134+0000", 2024-04-26T01:08:03.793 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001026117 2024-04-26T01:08:03.793 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.793 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.793 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.793 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446166+0000", 2024-04-26T01:08:03.793 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1208999999999999e-05 2024-04-26T01:08:03.793 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.793 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.793 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.793 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.793 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.794 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs3 33/30 ECSubWrite(tid=39, reqid=client.4521.0:130, at_version=33'3, trim_to=0'0, roll_forward_to=33'2))", 2024-04-26T01:08:03.794 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.461847+0000", 2024-04-26T01:08:03.794 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.2670645609999998, 2024-04-26T01:08:03.794 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0017972629999999999, 2024-04-26T01:08:03.794 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.794 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.794 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.794 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.794 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.794 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.794 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.461847+0000", 2024-04-26T01:08:03.794 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.794 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.794 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.795 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.795 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.461847+0000", 2024-04-26T01:08:03.795 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.795 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.795 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.461849+0000", 2024-04-26T01:08:03.795 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3939999999999999e-06 2024-04-26T01:08:03.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.795 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.795 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.462070+0000", 2024-04-26T01:08:03.795 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000220886 2024-04-26T01:08:03.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.796 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.462149+0000", 2024-04-26T01:08:03.796 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.9295000000000006e-05 2024-04-26T01:08:03.796 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.796 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.462161+0000", 2024-04-26T01:08:03.796 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1805e-05 2024-04-26T01:08:03.796 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.796 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.462262+0000", 2024-04-26T01:08:03.796 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010117500000000001 2024-04-26T01:08:03.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.797 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.797 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.797 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.462273+0000", 2024-04-26T01:08:03.797 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1411e-05 2024-04-26T01:08:03.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.797 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.797 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.797 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.463608+0000", 2024-04-26T01:08:03.797 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0013349429999999999 2024-04-26T01:08:03.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.797 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.797 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.797 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.463645+0000", 2024-04-26T01:08:03.798 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6353999999999998e-05 2024-04-26T01:08:03.798 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.798 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.798 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.798 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.798 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.798 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 33/30 ECSubWrite(tid=6, reqid=client.4521.0:136, at_version=33'5, trim_to=0'0, roll_forward_to=33'3))", 2024-04-26T01:08:03.798 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.469887+0000", 2024-04-26T01:08:03.798 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.2590249360000003, 2024-04-26T01:08:03.798 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0014679370000000001, 2024-04-26T01:08:03.798 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.798 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.798 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.799 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.469887+0000", 2024-04-26T01:08:03.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.469887+0000", 2024-04-26T01:08:03.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.469888+0000", 2024-04-26T01:08:03.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9100000000000002e-07 2024-04-26T01:08:03.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.800 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.800 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.470317+0000", 2024-04-26T01:08:03.800 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000429512 2024-04-26T01:08:03.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.800 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.800 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.470357+0000", 2024-04-26T01:08:03.800 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9978000000000002e-05 2024-04-26T01:08:03.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.800 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.800 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.470367+0000", 2024-04-26T01:08:03.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.5799999999999998e-06 2024-04-26T01:08:03.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.801 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.801 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.470427+0000", 2024-04-26T01:08:03.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0027e-05 2024-04-26T01:08:03.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.801 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.801 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.470433+0000", 2024-04-26T01:08:03.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6949999999999997e-06 2024-04-26T01:08:03.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.471333+0000", 2024-04-26T01:08:03.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00090048500000000002 2024-04-26T01:08:03.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.471355+0000", 2024-04-26T01:08:03.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1769000000000001e-05 2024-04-26T01:08:03.802 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.802 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.802 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.802 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 33/30 ECSubWrite(tid=34, reqid=client.4521.0:139, at_version=33'11, trim_to=0'0, roll_forward_to=33'10))", 2024-04-26T01:08:03.802 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.474229+0000", 2024-04-26T01:08:03.803 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.2546831090000001, 2024-04-26T01:08:03.803 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0014759319999999999, 2024-04-26T01:08:03.803 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.803 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.803 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.803 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.803 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.803 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.803 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.474229+0000", 2024-04-26T01:08:03.803 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.803 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.803 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.803 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.803 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.474229+0000", 2024-04-26T01:08:03.803 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.804 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.804 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.804 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.804 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.474230+0000", 2024-04-26T01:08:03.804 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9300000000000006e-07 2024-04-26T01:08:03.804 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.804 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.804 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.804 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.474582+0000", 2024-04-26T01:08:03.804 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00035193600000000001 2024-04-26T01:08:03.804 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.804 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.804 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.804 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.474625+0000", 2024-04-26T01:08:03.804 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3606000000000002e-05 2024-04-26T01:08:03.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.805 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.805 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.474634+0000", 2024-04-26T01:08:03.805 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9979999999999992e-06 2024-04-26T01:08:03.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.805 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.805 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.474701+0000", 2024-04-26T01:08:03.805 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6586000000000002e-05 2024-04-26T01:08:03.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.805 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.805 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.474711+0000", 2024-04-26T01:08:03.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.007e-05 2024-04-26T01:08:03.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.806 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.806 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.475676+0000", 2024-04-26T01:08:03.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00096522299999999997 2024-04-26T01:08:03.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.806 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.806 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.475705+0000", 2024-04-26T01:08:03.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8520000000000001e-05 2024-04-26T01:08:03.806 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.806 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.807 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.807 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.807 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 33/30 ECSubWrite(tid=46, reqid=client.4521.0:158, at_version=33'15, trim_to=0'0, roll_forward_to=33'12))", 2024-04-26T01:08:03.807 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.497963+0000", 2024-04-26T01:08:03.807 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.2309491970000002, 2024-04-26T01:08:03.807 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0014564980000000001, 2024-04-26T01:08:03.807 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.807 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.807 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.807 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.807 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.807 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.497963+0000", 2024-04-26T01:08:03.807 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.808 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.808 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.808 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.808 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.497963+0000", 2024-04-26T01:08:03.808 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.808 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.808 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.808 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.808 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.497963+0000", 2024-04-26T01:08:03.808 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1199999999999998e-07 2024-04-26T01:08:03.808 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.808 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.808 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.808 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.497986+0000", 2024-04-26T01:08:03.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2867999999999999e-05 2024-04-26T01:08:03.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.809 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.809 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.498080+0000", 2024-04-26T01:08:03.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4411000000000001e-05 2024-04-26T01:08:03.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.809 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.809 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.498087+0000", 2024-04-26T01:08:03.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9929999999999998e-06 2024-04-26T01:08:03.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.810 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.810 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.498586+0000", 2024-04-26T01:08:03.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00049807699999999998 2024-04-26T01:08:03.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.810 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.810 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.498594+0000", 2024-04-26T01:08:03.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4330000000000004e-06 2024-04-26T01:08:03.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.811 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.811 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.499387+0000", 2024-04-26T01:08:03.811 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00079311599999999996 2024-04-26T01:08:03.811 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.811 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.811 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.811 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.499419+0000", 2024-04-26T01:08:03.811 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.2187999999999997e-05 2024-04-26T01:08:03.811 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.811 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.811 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.811 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.811 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.811 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 33/30 ECSubWrite(tid=10, reqid=client.4521.0:171, at_version=33'8, trim_to=0'0, roll_forward_to=33'5))", 2024-04-26T01:08:03.812 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.522542+0000", 2024-04-26T01:08:03.812 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.2063697089999996, 2024-04-26T01:08:03.812 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0014466679999999999, 2024-04-26T01:08:03.812 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.812 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.812 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.812 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.812 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.812 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.812 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.522542+0000", 2024-04-26T01:08:03.812 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.812 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.812 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.812 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.812 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.522542+0000", 2024-04-26T01:08:03.813 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.813 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.813 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.522543+0000", 2024-04-26T01:08:03.813 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9299999999999997e-07 2024-04-26T01:08:03.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.813 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.813 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.522690+0000", 2024-04-26T01:08:03.813 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000146656 2024-04-26T01:08:03.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.813 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.522734+0000", 2024-04-26T01:08:03.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3952999999999997e-05 2024-04-26T01:08:03.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.522743+0000", 2024-04-26T01:08:03.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2879999999999998e-06 2024-04-26T01:08:03.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.522812+0000", 2024-04-26T01:08:03.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9003000000000003e-05 2024-04-26T01:08:03.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.522823+0000", 2024-04-26T01:08:03.815 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1413999999999999e-05 2024-04-26T01:08:03.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.523953+0000", 2024-04-26T01:08:03.815 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0011299179999999999 2024-04-26T01:08:03.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.523989+0000", 2024-04-26T01:08:03.815 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.5743e-05 2024-04-26T01:08:03.816 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.816 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.816 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.816 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.816 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.816 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 36/30 ECSubWrite(tid=16, reqid=client.4521.0:233, at_version=0'0, trim_to=0'0, roll_forward_to=36'11))", 2024-04-26T01:08:03.816 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.129868+0000", 2024-04-26T01:08:03.816 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.59904359600000001, 2024-04-26T01:08:03.816 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015585550000000001, 2024-04-26T01:08:03.816 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.816 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.816 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.816 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.816 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.816 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.817 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.129868+0000", 2024-04-26T01:08:03.817 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.817 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.817 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.129868+0000", 2024-04-26T01:08:03.817 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.817 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.817 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.129870+0000", 2024-04-26T01:08:03.817 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0600000000000002e-06 2024-04-26T01:08:03.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.817 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.129894+0000", 2024-04-26T01:08:03.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3932000000000001e-05 2024-04-26T01:08:03.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.129993+0000", 2024-04-26T01:08:03.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8152999999999994e-05 2024-04-26T01:08:03.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.130013+0000", 2024-04-26T01:08:03.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0896000000000001e-05 2024-04-26T01:08:03.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.819 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.130110+0000", 2024-04-26T01:08:03.819 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6475e-05 2024-04-26T01:08:03.819 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.819 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.130118+0000", 2024-04-26T01:08:03.819 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8420000000000005e-06 2024-04-26T01:08:03.819 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.819 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.131333+0000", 2024-04-26T01:08:03.819 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0012148650000000001 2024-04-26T01:08:03.820 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.820 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.820 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.820 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.131427+0000", 2024-04-26T01:08:03.820 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4332000000000004e-05 2024-04-26T01:08:03.820 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.820 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.820 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.820 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.820 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.820 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 37/30 ECSubWrite(tid=17, reqid=client.4521.0:243, at_version=37'13, trim_to=0'0, roll_forward_to=36'11))", 2024-04-26T01:08:03.820 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.423152+0000", 2024-04-26T01:08:03.820 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.30576041700000001, 2024-04-26T01:08:03.820 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027083010000000002, 2024-04-26T01:08:03.821 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:03.821 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:03.821 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:03.821 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:03.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.821 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:03.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423152+0000", 2024-04-26T01:08:03.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.821 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:03.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423152+0000", 2024-04-26T01:08:03.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:03.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:03.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423152+0000", 2024-04-26T01:08:03.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8900000000000001e-07 2024-04-26T01:08:03.822 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:03.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423668+0000", 2024-04-26T01:08:03.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00051661399999999998 2024-04-26T01:08:03.822 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:03.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423705+0000", 2024-04-26T01:08:03.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6186999999999998e-05 2024-04-26T01:08:03.823 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.823 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.823 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:03.823 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423714+0000", 2024-04-26T01:08:03.823 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0969999999999995e-06 2024-04-26T01:08:03.823 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.823 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.823 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:03.823 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423786+0000", 2024-04-26T01:08:03.823 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2013000000000003e-05 2024-04-26T01:08:03.823 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.823 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.823 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:03.823 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423796+0000", 2024-04-26T01:08:03.823 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0179e-05 2024-04-26T01:08:03.824 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.824 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.824 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:03.824 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.425806+0000", 2024-04-26T01:08:03.824 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0020104570000000002 2024-04-26T01:08:03.824 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:03.824 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:03.824 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:03.824 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.425860+0000", 2024-04-26T01:08:03.824 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3565e-05 2024-04-26T01:08:03.824 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.824 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.824 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.824 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:03.824 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:03.825 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:03.826 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:08:03.827 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:03.825+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:03.927 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:03.921+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:03.962 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:03.963 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:03.963 INFO:teuthology.orchestra.run.smithi012.stdout: "num_ops": 0 2024-04-26T01:08:03.963 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:03.977 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:08:04.065 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:04.061+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:04.131 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:04.132+0000 7f6879827640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25268) UID: 0 2024-04-26T01:08:04.135 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-26T01:08:04.135 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T01:08:04.135 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-04-26T01:08:04.135 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-26T01:08:04.135 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-26T01:08:04.136 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-04-26T01:08:04.141 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:04.141 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:04.141 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:08:04.141 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:08:04.141 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:04.155 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:08:04.240 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:04.237+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:04.268 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:04.268 INFO:teuthology.orchestra.run.smithi164.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-26T01:08:04.268 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:04.328 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:04.328 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:08:04.329 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:08:04.329 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:08:04.329 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.329 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.2s5 32/30 ECSubWrite(tid=157, reqid=client.4521.0:13, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:04.329 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.308314+0000", 2024-04-26T01:08:04.329 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.0176615470000003, 2024-04-26T01:08:04.329 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011288362999999999, 2024-04-26T01:08:04.329 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.329 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.329 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.329 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.329 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.329 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.330 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308314+0000", 2024-04-26T01:08:04.330 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.330 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.330 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.330 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308314+0000", 2024-04-26T01:08:04.330 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.330 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.330 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.330 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308315+0000", 2024-04-26T01:08:04.330 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.6599999999999995e-07 2024-04-26T01:08:04.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.331 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.331 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308467+0000", 2024-04-26T01:08:04.331 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000152249 2024-04-26T01:08:04.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.331 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.331 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308602+0000", 2024-04-26T01:08:04.331 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000135048 2024-04-26T01:08:04.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.331 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.331 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308653+0000", 2024-04-26T01:08:04.332 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0864e-05 2024-04-26T01:08:04.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.332 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.332 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.332 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308834+0000", 2024-04-26T01:08:04.332 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000180466 2024-04-26T01:08:04.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.332 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.332 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.332 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308857+0000", 2024-04-26T01:08:04.332 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2904e-05 2024-04-26T01:08:04.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.332 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.332 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.332 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.319506+0000", 2024-04-26T01:08:04.333 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010649141000000001 2024-04-26T01:08:04.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.333 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.333 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.333 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.319603+0000", 2024-04-26T01:08:04.333 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6924999999999998e-05 2024-04-26T01:08:04.333 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.333 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.333 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.333 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.333 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 32/30 ECSubWrite(tid=0, reqid=client.4521.0:16, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:04.333 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:04.333 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.0154432279999996, 2024-04-26T01:08:04.333 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012119314000000001, 2024-04-26T01:08:04.334 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.334 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.334 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.334 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.334 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.334 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:04.334 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.334 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.334 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:04.334 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.335 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.335 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:04.335 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.7000000000000004e-07 2024-04-26T01:08:04.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.335 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.335 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310733+0000", 2024-04-26T01:08:04.335 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019985599999999999 2024-04-26T01:08:04.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.335 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.335 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310792+0000", 2024-04-26T01:08:04.335 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.8894000000000002e-05 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310816+0000", 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4094e-05 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311022+0000", 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00020608599999999999 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311044+0000", 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1889999999999999e-05 2024-04-26T01:08:04.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.337 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.337 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322407+0000", 2024-04-26T01:08:04.337 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011362778 2024-04-26T01:08:04.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.337 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.337 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322652+0000", 2024-04-26T01:08:04.337 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00024504600000000001 2024-04-26T01:08:04.337 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.337 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.337 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.337 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.es2 33/30 ECSubWrite(tid=0, reqid=client.4521.0:22, at_version=33'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:04.338 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.311344+0000", 2024-04-26T01:08:04.338 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.0146317140000001, 2024-04-26T01:08:04.338 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010979113, 2024-04-26T01:08:04.338 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.338 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.338 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.338 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.338 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.338 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.338 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311344+0000", 2024-04-26T01:08:04.338 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.338 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.338 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.339 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.339 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311344+0000", 2024-04-26T01:08:04.339 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.339 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.339 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.339 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.339 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311351+0000", 2024-04-26T01:08:04.339 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1040000000000001e-06 2024-04-26T01:08:04.339 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.339 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.339 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.339 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311496+0000", 2024-04-26T01:08:04.339 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000144852 2024-04-26T01:08:04.339 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.339 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.340 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.340 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311639+0000", 2024-04-26T01:08:04.340 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014267800000000001 2024-04-26T01:08:04.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.340 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.340 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.340 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311663+0000", 2024-04-26T01:08:04.340 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4090000000000001e-05 2024-04-26T01:08:04.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.340 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.340 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.340 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311836+0000", 2024-04-26T01:08:04.340 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000173349 2024-04-26T01:08:04.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.341 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.341 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.341 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311859+0000", 2024-04-26T01:08:04.341 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2504e-05 2024-04-26T01:08:04.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.341 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.341 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.341 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322193+0000", 2024-04-26T01:08:04.341 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010334157 2024-04-26T01:08:04.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.342 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.342 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.342 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322323+0000", 2024-04-26T01:08:04.342 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013037899999999999 2024-04-26T01:08:04.342 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.342 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.342 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.342 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.342 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.342 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.8s3 32/30 ECSubWrite(tid=1, reqid=client.4521.0:8, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:04.342 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.312999+0000", 2024-04-26T01:08:04.342 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.0129772519999998, 2024-04-26T01:08:04.342 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0094850960000000002, 2024-04-26T01:08:04.342 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.343 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.343 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.343 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.343 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.343 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.343 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312999+0000", 2024-04-26T01:08:04.343 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.343 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.343 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.343 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.343 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312999+0000", 2024-04-26T01:08:04.343 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.343 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.344 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.344 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.344 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313001+0000", 2024-04-26T01:08:04.344 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.21e-06 2024-04-26T01:08:04.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.344 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.344 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.344 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313076+0000", 2024-04-26T01:08:04.344 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.5525000000000006e-05 2024-04-26T01:08:04.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.344 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.344 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.344 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313194+0000", 2024-04-26T01:08:04.344 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000118004 2024-04-26T01:08:04.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.345 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.345 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.345 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313238+0000", 2024-04-26T01:08:04.345 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3523999999999999e-05 2024-04-26T01:08:04.345 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.345 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.345 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.345 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313570+0000", 2024-04-26T01:08:04.345 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033178100000000001 2024-04-26T01:08:04.345 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.345 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.345 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.346 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313595+0000", 2024-04-26T01:08:04.346 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5181000000000002e-05 2024-04-26T01:08:04.346 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.346 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.346 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.346 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322175+0000", 2024-04-26T01:08:04.346 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0085803070000000006 2024-04-26T01:08:04.346 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.346 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.346 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.346 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322484+0000", 2024-04-26T01:08:04.346 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000308564 2024-04-26T01:08:04.346 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.346 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.347 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.347 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.347 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.347 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.es2 33/30 ECSubWrite(tid=1, reqid=client.4521.0:23, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:04.347 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.313493+0000", 2024-04-26T01:08:04.347 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.0124832189999999, 2024-04-26T01:08:04.347 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0089452720000000006, 2024-04-26T01:08:04.347 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.347 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.347 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.347 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.347 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.347 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.348 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313493+0000", 2024-04-26T01:08:04.348 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.348 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.348 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.348 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.348 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313493+0000", 2024-04-26T01:08:04.348 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.348 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.348 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.348 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.348 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313494+0000", 2024-04-26T01:08:04.348 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5039999999999999e-06 2024-04-26T01:08:04.348 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.348 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.348 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.349 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313593+0000", 2024-04-26T01:08:04.349 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.9355e-05 2024-04-26T01:08:04.349 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.349 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.349 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.349 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313756+0000", 2024-04-26T01:08:04.349 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000162762 2024-04-26T01:08:04.349 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.349 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.349 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.349 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313781+0000", 2024-04-26T01:08:04.349 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4893e-05 2024-04-26T01:08:04.349 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.349 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.349 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.350 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313944+0000", 2024-04-26T01:08:04.350 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016297600000000001 2024-04-26T01:08:04.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.350 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.350 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.350 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313968+0000", 2024-04-26T01:08:04.350 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3986999999999999e-05 2024-04-26T01:08:04.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.350 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.350 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.350 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322340+0000", 2024-04-26T01:08:04.350 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0083720840000000001 2024-04-26T01:08:04.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.350 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.351 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.351 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322438+0000", 2024-04-26T01:08:04.351 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7711e-05 2024-04-26T01:08:04.351 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.351 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.351 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.351 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.351 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.351 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.2s5 32/30 ECSubWrite(tid=158, reqid=client.4521.0:14, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:04.351 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.317021+0000", 2024-04-26T01:08:04.351 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.0089544320000003, 2024-04-26T01:08:04.351 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0053200579999999999, 2024-04-26T01:08:04.351 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.351 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.351 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317021+0000", 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317021+0000", 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317024+0000", 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6400000000000001e-06 2024-04-26T01:08:04.352 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.353 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.353 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.353 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317075+0000", 2024-04-26T01:08:04.353 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0680999999999998e-05 2024-04-26T01:08:04.353 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.353 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.353 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.353 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317190+0000", 2024-04-26T01:08:04.353 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000114998 2024-04-26T01:08:04.353 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.353 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.353 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.353 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317213+0000", 2024-04-26T01:08:04.353 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2827e-05 2024-04-26T01:08:04.353 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.354 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.354 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.354 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317879+0000", 2024-04-26T01:08:04.354 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00066639300000000004 2024-04-26T01:08:04.354 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.354 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.354 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.354 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317911+0000", 2024-04-26T01:08:04.354 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.2400000000000001e-05 2024-04-26T01:08:04.354 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.354 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.354 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.354 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322261+0000", 2024-04-26T01:08:04.354 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.004349594 2024-04-26T01:08:04.355 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.355 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.355 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.355 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322341+0000", 2024-04-26T01:08:04.355 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.0525000000000006e-05 2024-04-26T01:08:04.355 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.355 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.355 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.355 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.355 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.355 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.es2 33/30 ECSubWrite(tid=2, reqid=client.4521.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=33'2))", 2024-04-26T01:08:04.355 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.323780+0000", 2024-04-26T01:08:04.355 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.0021955790000003, 2024-04-26T01:08:04.355 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0055927879999999996, 2024-04-26T01:08:04.355 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.356 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.356 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.356 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.356 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.356 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.356 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323780+0000", 2024-04-26T01:08:04.356 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.356 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.356 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.356 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.356 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323780+0000", 2024-04-26T01:08:04.356 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.356 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.356 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.356 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.357 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323782+0000", 2024-04-26T01:08:04.357 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.717e-06 2024-04-26T01:08:04.357 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.357 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.357 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.357 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323811+0000", 2024-04-26T01:08:04.357 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8634000000000001e-05 2024-04-26T01:08:04.357 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.357 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.357 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.357 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323899+0000", 2024-04-26T01:08:04.357 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.8015000000000003e-05 2024-04-26T01:08:04.357 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.357 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.358 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.358 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324040+0000", 2024-04-26T01:08:04.358 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000141898 2024-04-26T01:08:04.358 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.358 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.358 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.358 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324324+0000", 2024-04-26T01:08:04.358 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00028301699999999999 2024-04-26T01:08:04.358 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.358 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.358 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.358 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324347+0000", 2024-04-26T01:08:04.358 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3654000000000001e-05 2024-04-26T01:08:04.358 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.359 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.359 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.359 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.329290+0000", 2024-04-26T01:08:04.359 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0049432449999999998 2024-04-26T01:08:04.359 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.359 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.359 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.359 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.329373+0000", 2024-04-26T01:08:04.359 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.2608000000000006e-05 2024-04-26T01:08:04.359 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.359 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.359 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.359 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.359 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.360 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 33/30 ECSubWrite(tid=1, reqid=client.4521.0:16, at_version=0'0, trim_to=0'0, roll_forward_to=32'1))", 2024-04-26T01:08:04.360 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.325324+0000", 2024-04-26T01:08:04.360 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.0006521279999996, 2024-04-26T01:08:04.360 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0040649509999999998, 2024-04-26T01:08:04.360 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.360 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.360 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.360 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.360 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.360 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.360 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325324+0000", 2024-04-26T01:08:04.360 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.360 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.360 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.360 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.361 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325324+0000", 2024-04-26T01:08:04.361 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.361 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.361 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.361 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.361 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325326+0000", 2024-04-26T01:08:04.361 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8339999999999999e-06 2024-04-26T01:08:04.361 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.361 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.361 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.361 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325354+0000", 2024-04-26T01:08:04.361 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8853e-05 2024-04-26T01:08:04.361 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.361 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325444+0000", 2024-04-26T01:08:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.9258000000000005e-05 2024-04-26T01:08:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325467+0000", 2024-04-26T01:08:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3499999999999999e-05 2024-04-26T01:08:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325658+0000", 2024-04-26T01:08:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019086600000000001 2024-04-26T01:08:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.363 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.363 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325680+0000", 2024-04-26T01:08:04.363 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2096999999999999e-05 2024-04-26T01:08:04.363 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.363 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.363 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.363 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.329307+0000", 2024-04-26T01:08:04.363 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.003627178 2024-04-26T01:08:04.363 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.363 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.363 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.363 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.329389+0000", 2024-04-26T01:08:04.363 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1365000000000004e-05 2024-04-26T01:08:04.364 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.364 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.364 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.364 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.364 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.364 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.8s3 33/30 ECSubWrite(tid=2, reqid=client.4521.0:8, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:04.364 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.326417+0000", 2024-04-26T01:08:04.364 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.9995587500000003, 2024-04-26T01:08:04.364 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0057806029999999996, 2024-04-26T01:08:04.364 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.364 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.364 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.364 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.364 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.364 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.365 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326417+0000", 2024-04-26T01:08:04.365 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.365 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.365 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.365 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.365 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326417+0000", 2024-04-26T01:08:04.365 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.365 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.365 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.365 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.365 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326419+0000", 2024-04-26T01:08:04.365 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.753e-06 2024-04-26T01:08:04.365 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.365 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.366 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.366 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326479+0000", 2024-04-26T01:08:04.366 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.9800999999999999e-05 2024-04-26T01:08:04.366 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.366 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.366 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.366 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326579+0000", 2024-04-26T01:08:04.366 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.9970999999999995e-05 2024-04-26T01:08:04.366 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.366 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.366 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.367 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326602+0000", 2024-04-26T01:08:04.367 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3234000000000002e-05 2024-04-26T01:08:04.367 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.367 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.367 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.367 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327273+0000", 2024-04-26T01:08:04.367 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00067146199999999995 2024-04-26T01:08:04.367 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.367 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.367 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.367 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327298+0000", 2024-04-26T01:08:04.367 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4763999999999999e-05 2024-04-26T01:08:04.367 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.367 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.368 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.368 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.331893+0000", 2024-04-26T01:08:04.368 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0045952379999999998 2024-04-26T01:08:04.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.368 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.368 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.368 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.332198+0000", 2024-04-26T01:08:04.368 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00030437999999999998 2024-04-26T01:08:04.368 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.368 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.368 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.368 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.368 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.2s5 32/30 ECSubWrite(tid=159, reqid=client.4521.0:31, at_version=32'3, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:04.369 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.328602+0000", 2024-04-26T01:08:04.369 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.9973742379999999, 2024-04-26T01:08:04.369 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0043952720000000004, 2024-04-26T01:08:04.369 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.369 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.369 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.369 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.369 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.369 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.369 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328602+0000", 2024-04-26T01:08:04.369 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.369 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.370 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.370 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.370 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328602+0000", 2024-04-26T01:08:04.370 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.370 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.370 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.370 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328602+0000", 2024-04-26T01:08:04.370 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.9699999999999995e-07 2024-04-26T01:08:04.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.370 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.370 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.370 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328921+0000", 2024-04-26T01:08:04.371 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00031870100000000001 2024-04-26T01:08:04.371 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.371 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.371 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.371 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.329110+0000", 2024-04-26T01:08:04.371 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00018928599999999999 2024-04-26T01:08:04.371 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.371 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.371 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.371 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.329133+0000", 2024-04-26T01:08:04.371 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2716000000000001e-05 2024-04-26T01:08:04.371 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.371 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.371 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.372 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.329360+0000", 2024-04-26T01:08:04.372 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000226703 2024-04-26T01:08:04.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.372 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.372 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.372 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.329389+0000", 2024-04-26T01:08:04.372 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9601e-05 2024-04-26T01:08:04.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.372 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.372 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.372 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.332861+0000", 2024-04-26T01:08:04.372 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0034714160000000002 2024-04-26T01:08:04.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.372 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.372 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.373 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.332997+0000", 2024-04-26T01:08:04.373 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000136052 2024-04-26T01:08:04.373 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.373 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.373 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.373 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.373 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.373 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.2s5 33/30 ECSubWrite(tid=160, reqid=client.4521.0:32, at_version=33'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:04.373 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.328935+0000", 2024-04-26T01:08:04.373 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.9970411129999999, 2024-04-26T01:08:04.373 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0041303249999999998, 2024-04-26T01:08:04.373 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.373 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.373 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.374 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.374 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.374 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328935+0000", 2024-04-26T01:08:04.374 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.374 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.374 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.374 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328935+0000", 2024-04-26T01:08:04.374 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.374 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.374 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.374 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328936+0000", 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1069999999999999e-06 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328989+0000", 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.2957000000000001e-05 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.329201+0000", 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00021201600000000001 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.329236+0000", 2024-04-26T01:08:04.375 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.5194000000000002e-05 2024-04-26T01:08:04.376 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.376 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.376 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.376 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.330762+0000", 2024-04-26T01:08:04.376 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00152619 2024-04-26T01:08:04.376 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.376 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.376 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.376 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.330788+0000", 2024-04-26T01:08:04.376 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5457e-05 2024-04-26T01:08:04.376 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.376 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.376 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.376 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.333006+0000", 2024-04-26T01:08:04.377 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0022186929999999999 2024-04-26T01:08:04.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.377 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.377 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.377 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.333065+0000", 2024-04-26T01:08:04.377 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.8711000000000001e-05 2024-04-26T01:08:04.377 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.377 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.377 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.377 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.377 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.6s2 33/30 ECSubWrite(tid=1, reqid=client.4521.0:44, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:04.377 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.349917+0000", 2024-04-26T01:08:04.377 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.9760591249999999, 2024-04-26T01:08:04.378 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0052687769999999997, 2024-04-26T01:08:04.378 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.378 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.378 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.378 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.378 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.378 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.378 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.349917+0000", 2024-04-26T01:08:04.378 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.378 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.378 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.378 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.378 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.349917+0000", 2024-04-26T01:08:04.378 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.378 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.379 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.379 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.379 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.349918+0000", 2024-04-26T01:08:04.379 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.607e-06 2024-04-26T01:08:04.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.379 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.379 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.379 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.349961+0000", 2024-04-26T01:08:04.379 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.2870000000000001e-05 2024-04-26T01:08:04.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.379 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.379 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.379 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350046+0000", 2024-04-26T01:08:04.379 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.4957999999999995e-05 2024-04-26T01:08:04.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.380 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.380 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.380 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350069+0000", 2024-04-26T01:08:04.380 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2574000000000001e-05 2024-04-26T01:08:04.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.380 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.380 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.380 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350881+0000", 2024-04-26T01:08:04.380 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00081191100000000001 2024-04-26T01:08:04.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.380 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.380 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.381 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350900+0000", 2024-04-26T01:08:04.381 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9559999999999999e-05 2024-04-26T01:08:04.381 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.381 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.381 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.381 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.355103+0000", 2024-04-26T01:08:04.381 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0042028559999999996 2024-04-26T01:08:04.381 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.381 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.381 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.381 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.355185+0000", 2024-04-26T01:08:04.381 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.2441000000000006e-05 2024-04-26T01:08:04.381 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.381 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.382 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.382 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.382 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.382 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.8s3 33/30 ECSubWrite(tid=5, reqid=client.4521.0:50, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:04.382 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.350722+0000", 2024-04-26T01:08:04.382 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.9752536740000002, 2024-04-26T01:08:04.382 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0042213490000000001, 2024-04-26T01:08:04.382 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.382 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.382 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.382 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.382 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.382 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.383 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350722+0000", 2024-04-26T01:08:04.383 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.383 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.383 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.383 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.383 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350722+0000", 2024-04-26T01:08:04.383 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.383 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.383 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.383 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.383 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350723+0000", 2024-04-26T01:08:04.383 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.248e-06 2024-04-26T01:08:04.383 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.383 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.383 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.384 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350751+0000", 2024-04-26T01:08:04.384 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7257999999999999e-05 2024-04-26T01:08:04.384 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.384 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.384 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.384 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350820+0000", 2024-04-26T01:08:04.384 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.9035999999999994e-05 2024-04-26T01:08:04.384 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.384 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.384 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.384 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350838+0000", 2024-04-26T01:08:04.384 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8031999999999999e-05 2024-04-26T01:08:04.384 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.384 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.384 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.385 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350956+0000", 2024-04-26T01:08:04.385 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000118377 2024-04-26T01:08:04.385 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.385 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.385 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.385 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350975+0000", 2024-04-26T01:08:04.385 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9174999999999999e-05 2024-04-26T01:08:04.385 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.385 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.385 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.385 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.354859+0000", 2024-04-26T01:08:04.385 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0038841850000000001 2024-04-26T01:08:04.385 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.386 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.386 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.386 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.354943+0000", 2024-04-26T01:08:04.386 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.4037999999999997e-05 2024-04-26T01:08:04.386 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.386 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.386 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.386 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.386 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.386 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s3 33/30 ECSubWrite(tid=1, reqid=client.4521.0:104, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:04.386 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.427663+0000", 2024-04-26T01:08:04.386 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.8983125520000002, 2024-04-26T01:08:04.386 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0040135759999999996, 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.427663+0000", 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.427663+0000", 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.387 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.388 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.427665+0000", 2024-04-26T01:08:04.388 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8700000000000001e-06 2024-04-26T01:08:04.388 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.388 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.388 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.388 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.427696+0000", 2024-04-26T01:08:04.388 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.0985000000000002e-05 2024-04-26T01:08:04.388 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.388 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.388 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.389 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.427779+0000", 2024-04-26T01:08:04.389 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.2646999999999997e-05 2024-04-26T01:08:04.389 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.389 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.389 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.389 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.427799+0000", 2024-04-26T01:08:04.389 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0285999999999999e-05 2024-04-26T01:08:04.389 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.389 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.389 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.389 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.428448+0000", 2024-04-26T01:08:04.389 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00064852699999999998 2024-04-26T01:08:04.389 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.389 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.390 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.390 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.428460+0000", 2024-04-26T01:08:04.390 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2261e-05 2024-04-26T01:08:04.390 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.390 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.390 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.390 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.431606+0000", 2024-04-26T01:08:04.390 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0031464959999999999 2024-04-26T01:08:04.390 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.390 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.390 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.390 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.431677+0000", 2024-04-26T01:08:04.390 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.0504000000000001e-05 2024-04-26T01:08:04.391 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.391 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.391 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.391 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.391 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.391 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s3 33/30 ECSubWrite(tid=2, reqid=client.4521.0:104, at_version=0'0, trim_to=0'0, roll_forward_to=33'2))", 2024-04-26T01:08:04.391 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.437607+0000", 2024-04-26T01:08:04.391 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.8883686810000002, 2024-04-26T01:08:04.391 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.003989851, 2024-04-26T01:08:04.391 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.391 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.391 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.391 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.391 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.391 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.392 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.437607+0000", 2024-04-26T01:08:04.392 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.392 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.392 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.392 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.392 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.437607+0000", 2024-04-26T01:08:04.392 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.392 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.392 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.392 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.392 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.437609+0000", 2024-04-26T01:08:04.392 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7320000000000001e-06 2024-04-26T01:08:04.392 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.392 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.392 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.393 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.437630+0000", 2024-04-26T01:08:04.393 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1444e-05 2024-04-26T01:08:04.393 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.393 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.393 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.393 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.437725+0000", 2024-04-26T01:08:04.393 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.4689000000000001e-05 2024-04-26T01:08:04.393 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.393 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.393 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.393 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.437741+0000", 2024-04-26T01:08:04.393 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6269999999999998e-05 2024-04-26T01:08:04.393 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.394 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.394 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.394 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.437853+0000", 2024-04-26T01:08:04.394 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00011134 2024-04-26T01:08:04.394 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.394 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.394 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.394 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.437870+0000", 2024-04-26T01:08:04.394 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7714999999999999e-05 2024-04-26T01:08:04.394 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.394 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.394 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.394 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.441530+0000", 2024-04-26T01:08:04.395 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0036592439999999999 2024-04-26T01:08:04.395 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.395 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.395 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.395 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.441597+0000", 2024-04-26T01:08:04.395 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.7416999999999995e-05 2024-04-26T01:08:04.395 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.395 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.395 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.395 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.395 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.395 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/30 ECSubWrite(tid=17, reqid=client.4521.0:224, at_version=36'10, trim_to=0'0, roll_forward_to=33'8))", 2024-04-26T01:08:04.395 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:02.430740+0000", 2024-04-26T01:08:04.395 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 1.895236087, 2024-04-26T01:08:04.396 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0054526749999999997, 2024-04-26T01:08:04.396 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.396 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.396 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.396 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.396 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.396 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.396 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.430740+0000", 2024-04-26T01:08:04.396 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.396 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.396 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.396 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.396 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.430740+0000", 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.430740+0000", 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.4400000000000001e-07 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.430878+0000", 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013799099999999999 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.430978+0000", 2024-04-26T01:08:04.397 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000100188 2024-04-26T01:08:04.398 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.398 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.398 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.398 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.431004+0000", 2024-04-26T01:08:04.398 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5490000000000002e-05 2024-04-26T01:08:04.398 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.398 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.398 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.398 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.431197+0000", 2024-04-26T01:08:04.398 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019329300000000001 2024-04-26T01:08:04.398 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.398 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.398 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.399 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.431217+0000", 2024-04-26T01:08:04.399 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.02e-05 2024-04-26T01:08:04.399 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.399 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.399 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.399 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.436087+0000", 2024-04-26T01:08:04.399 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0048693039999999996 2024-04-26T01:08:04.399 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.399 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.399 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.399 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.436192+0000", 2024-04-26T01:08:04.399 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000105765 2024-04-26T01:08:04.399 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.399 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.399 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.400 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.400 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.400 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.2s5 36/30 ECSubWrite(tid=184, reqid=client.4521.0:225, at_version=36'13, trim_to=0'0, roll_forward_to=33'10))", 2024-04-26T01:08:04.400 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:02.437781+0000", 2024-04-26T01:08:04.400 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 1.8881953010000001, 2024-04-26T01:08:04.400 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0053170939999999996, 2024-04-26T01:08:04.400 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.400 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.400 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.400 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.400 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.400 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.400 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.437781+0000", 2024-04-26T01:08:04.400 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.400 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.401 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.401 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.401 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.437781+0000", 2024-04-26T01:08:04.401 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.401 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.401 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.401 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.401 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.437781+0000", 2024-04-26T01:08:04.401 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2299999999999999e-07 2024-04-26T01:08:04.401 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.401 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.401 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.401 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.437843+0000", 2024-04-26T01:08:04.401 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.1851000000000001e-05 2024-04-26T01:08:04.402 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.402 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.402 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.402 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.437990+0000", 2024-04-26T01:08:04.402 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014688799999999999 2024-04-26T01:08:04.402 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.402 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.402 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.402 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.438017+0000", 2024-04-26T01:08:04.402 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7146999999999999e-05 2024-04-26T01:08:04.402 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.402 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.402 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.402 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.438295+0000", 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00027786999999999999 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.438321+0000", 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5431000000000001e-05 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.443000+0000", 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0046792759999999996 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.443098+0000", 2024-04-26T01:08:04.403 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7707999999999994e-05 2024-04-26T01:08:04.404 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.404 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.404 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.404 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.404 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.404 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.2s5 36/30 ECSubWrite(tid=185, reqid=client.4521.0:225, at_version=0'0, trim_to=0'0, roll_forward_to=36'13))", 2024-04-26T01:08:04.404 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:02.445957+0000", 2024-04-26T01:08:04.404 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 1.8800192149999999, 2024-04-26T01:08:04.404 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0052070720000000001, 2024-04-26T01:08:04.404 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.404 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.404 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.404 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.404 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.404 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.405 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.445957+0000", 2024-04-26T01:08:04.405 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.405 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.405 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.405 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.405 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.445957+0000", 2024-04-26T01:08:04.405 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.405 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.405 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.405 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.405 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.445958+0000", 2024-04-26T01:08:04.405 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6130000000000001e-06 2024-04-26T01:08:04.405 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.405 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.406 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.406 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.445979+0000", 2024-04-26T01:08:04.406 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0893000000000001e-05 2024-04-26T01:08:04.406 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.406 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.406 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.406 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.446058+0000", 2024-04-26T01:08:04.406 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8450999999999998e-05 2024-04-26T01:08:04.406 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.406 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.406 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.406 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.446073+0000", 2024-04-26T01:08:04.406 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5167000000000001e-05 2024-04-26T01:08:04.406 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.407 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.407 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.407 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.446249+0000", 2024-04-26T01:08:04.407 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017641900000000001 2024-04-26T01:08:04.407 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.407 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.407 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.407 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.446281+0000", 2024-04-26T01:08:04.407 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.2123999999999999e-05 2024-04-26T01:08:04.407 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.407 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.407 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.407 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.451082+0000", 2024-04-26T01:08:04.407 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0048012009999999997 2024-04-26T01:08:04.408 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.408 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.408 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.408 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.451164+0000", 2024-04-26T01:08:04.408 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1204000000000004e-05 2024-04-26T01:08:04.408 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.408 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.408 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.408 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.408 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.408 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.es2 36/30 ECSubWrite(tid=15, reqid=client.4521.0:233, at_version=36'11, trim_to=0'0, roll_forward_to=33'9))", 2024-04-26T01:08:04.409 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:02.479576+0000", 2024-04-26T01:08:04.409 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 1.8463993830000001, 2024-04-26T01:08:04.409 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0041685339999999998, 2024-04-26T01:08:04.409 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.409 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.409 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.409 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.409 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.409 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.409 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.479576+0000", 2024-04-26T01:08:04.409 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.409 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.409 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.410 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.410 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.479576+0000", 2024-04-26T01:08:04.410 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.410 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.410 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.410 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.410 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.479577+0000", 2024-04-26T01:08:04.410 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.2699999999999999e-07 2024-04-26T01:08:04.410 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.410 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.410 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.410 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.480333+0000", 2024-04-26T01:08:04.410 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00075548200000000003 2024-04-26T01:08:04.411 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.411 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.411 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.411 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.480429+0000", 2024-04-26T01:08:04.411 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6460999999999997e-05 2024-04-26T01:08:04.411 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.411 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.411 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.411 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.480453+0000", 2024-04-26T01:08:04.411 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3997000000000001e-05 2024-04-26T01:08:04.411 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.411 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.411 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.411 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.480627+0000", 2024-04-26T01:08:04.412 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017351500000000001 2024-04-26T01:08:04.412 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.412 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.412 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.412 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.480656+0000", 2024-04-26T01:08:04.412 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9609999999999999e-05 2024-04-26T01:08:04.412 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.412 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.412 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.412 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.483649+0000", 2024-04-26T01:08:04.412 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002993037 2024-04-26T01:08:04.412 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.412 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.412 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.413 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.483745+0000", 2024-04-26T01:08:04.413 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5804999999999995e-05 2024-04-26T01:08:04.413 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.413 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.413 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.413 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.413 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.413 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.es2 37/30 ECSubWrite(tid=17, reqid=client.4521.0:243, at_version=37'13, trim_to=0'0, roll_forward_to=36'11))", 2024-04-26T01:08:04.413 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:03.423578+0000", 2024-04-26T01:08:04.413 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 0.90239790399999997, 2024-04-26T01:08:04.413 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.003752586, 2024-04-26T01:08:04.413 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.413 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.413 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.413 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.414 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.414 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.414 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:03.423578+0000", 2024-04-26T01:08:04.414 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.414 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.414 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.414 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.414 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:03.423578+0000", 2024-04-26T01:08:04.414 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.414 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.414 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.414 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.414 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:03.423581+0000", 2024-04-26T01:08:04.414 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.467e-06 2024-04-26T01:08:04.414 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.415 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.415 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.415 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:03.424190+0000", 2024-04-26T01:08:04.415 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00060884700000000003 2024-04-26T01:08:04.415 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.415 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.415 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.415 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:03.424280+0000", 2024-04-26T01:08:04.415 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.0017000000000002e-05 2024-04-26T01:08:04.415 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.415 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.415 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.415 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:03.424303+0000", 2024-04-26T01:08:04.415 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3181e-05 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:03.424429+0000", 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012570499999999999 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:03.424445+0000", 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5650000000000001e-05 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:03.427234+0000", 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0027894719999999999 2024-04-26T01:08:04.416 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.417 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.417 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.417 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:03.427330+0000", 2024-04-26T01:08:04.417 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6247000000000002e-05 2024-04-26T01:08:04.417 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.417 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.417 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.417 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.417 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.417 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:04.420 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:08:04.421 INFO:tasks.rados.rados.0.smithi164.stdout:99: done (6 left) 2024-04-26T01:08:04.421 INFO:tasks.rados.rados.0.smithi164.stdout:100: read oid 14 snap -1 2024-04-26T01:08:04.421 INFO:tasks.rados.rados.0.smithi164.stdout:100: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:08:04.421 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 47 current snap is 3 2024-04-26T01:08:04.421 INFO:tasks.rados.rados.0.smithi164.stdout:93: finishing write tid 1 to smithi16428812-46 2024-04-26T01:08:04.422 INFO:tasks.rados.rados.0.smithi164.stdout:96: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-26T01:08:04.422 INFO:tasks.rados.rados.0.smithi164.stdout:95: finishing rollback tid 0 to smithi16428812-10 2024-04-26T01:08:04.422 INFO:tasks.rados.rados.0.smithi164.stdout:95: finishing rollback tid 1 to smithi16428812-10 2024-04-26T01:08:04.422 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 34 v 13 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-26T01:08:04.422 INFO:tasks.rados.rados.0.smithi164.stdout:94: finishing copy_from racing read to smithi16428812-45 2024-04-26T01:08:04.422 INFO:tasks.rados.rados.0.smithi164.stdout:93: finishing write tid 2 to smithi16428812-46 2024-04-26T01:08:04.422 INFO:tasks.rados.rados.0.smithi164.stdout:93: finishing write tid 3 to smithi16428812-46 2024-04-26T01:08:04.422 INFO:tasks.rados.rados.0.smithi164.stdout:101: seq_num 65 ranges {786432=720896,1507328=376832} 2024-04-26T01:08:04.422 INFO:tasks.rados.rados.0.smithi164.stdout:101: writing smithi16428812-47 from 786432 to 1507328 tid 1 2024-04-26T01:08:04.422 INFO:tasks.rados.rados.0.smithi164.stdout:101: writing smithi16428812-47 from 1507328 to 1884160 tid 2 2024-04-26T01:08:04.422 INFO:tasks.rados.rados.0.smithi164.stdout:96: done (7 left) 2024-04-26T01:08:04.422 INFO:tasks.rados.rados.0.smithi164.stdout:98: done (6 left) 2024-04-26T01:08:04.422 INFO:tasks.rados.rados.0.smithi164.stdout:102: copy_from oid 26 from oid 50 current snap is 3 2024-04-26T01:08:04.422 INFO:tasks.rados.rados.0.smithi164.stdout:94: finishing copy_from to smithi16428812-45 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 45 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:93: finishing write tid 4 to smithi16428812-46 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:93: oid 46 updating version 0 to 15 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:93: oid 46 version 15 is already newer than 14 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:93: oid 46 version 15 is already newer than 13 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 46 v 15 (ObjNum 64 snap 3 seq_num 64) dirty exists 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:93: left oid 46 (ObjNum 64 snap 3 seq_num 64) 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:97: finishing rollback tid 0 to smithi16428812-8 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:97: finishing rollback tid 1 to smithi16428812-8 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:95: finishing rollback tid 2 to smithi16428812-10 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 10 v 17 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:97: finishing rollback tid 2 to smithi16428812-8 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 8 v 19 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:93: done (6 left) 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:94: done (5 left) 2024-04-26T01:08:04.423 INFO:tasks.rados.rados.0.smithi164.stdout:95: done (4 left) 2024-04-26T01:08:04.424 INFO:tasks.rados.rados.0.smithi164.stdout:97: done (3 left) 2024-04-26T01:08:04.424 INFO:tasks.rados.rados.0.smithi164.stdout:103: read oid 19 snap 0 2024-04-26T01:08:04.424 INFO:tasks.rados.rados.0.smithi164.stdout:103: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-26T01:08:04.424 INFO:tasks.rados.rados.0.smithi164.stdout:104: copy_from oid 27 from oid 45 current snap is 3 2024-04-26T01:08:04.424 INFO:tasks.rados.rados.0.smithi164.stdout:105: read oid 24 snap -1 2024-04-26T01:08:04.424 INFO:tasks.rados.rados.0.smithi164.stdout:105: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-26T01:08:04.424 INFO:tasks.rados.rados.0.smithi164.stdout:106: rollback oid 25 current snap is 3 2024-04-26T01:08:04.424 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 25 to 0 2024-04-26T01:08:04.424 INFO:tasks.rados.rados.0.smithi164.stdout:100: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:08:04.424 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:04.417+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:04.490 INFO:tasks.rados.rados.0.smithi164.stdout:101: finishing write tid 1 to smithi16428812-47 2024-04-26T01:08:04.490 INFO:tasks.rados.rados.0.smithi164.stdout:100: done (6 left) 2024-04-26T01:08:04.490 INFO:tasks.rados.rados.0.smithi164.stdout:107: delete oid 37 current snap is 3 2024-04-26T01:08:04.490 INFO:tasks.rados.rados.0.smithi164.stdout:101: finishing write tid 2 to smithi16428812-47 2024-04-26T01:08:04.490 INFO:tasks.rados.rados.0.smithi164.stdout:101: finishing write tid 3 to smithi16428812-47 2024-04-26T01:08:04.490 INFO:tasks.rados.rados.0.smithi164.stdout:103: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-26T01:08:04.501 INFO:tasks.rados.rados.0.smithi164.stdout:105: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-26T01:08:04.519 INFO:tasks.rados.rados.0.smithi164.stdout:106: finishing rollback tid 0 to smithi16428812-25 2024-04-26T01:08:04.519 INFO:tasks.rados.rados.0.smithi164.stdout:103: done (6 left) 2024-04-26T01:08:04.519 INFO:tasks.rados.rados.0.smithi164.stdout:105: done (5 left) 2024-04-26T01:08:04.519 INFO:tasks.rados.rados.0.smithi164.stdout:107: done (4 left) 2024-04-26T01:08:04.520 INFO:tasks.rados.rados.0.smithi164.stdout:108: read oid 22 snap 0 2024-04-26T01:08:04.520 INFO:tasks.rados.rados.0.smithi164.stdout:108: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-26T01:08:04.520 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 35 current snap is 3 2024-04-26T01:08:04.520 INFO:tasks.rados.rados.0.smithi164.stdout:101: finishing write tid 4 to smithi16428812-47 2024-04-26T01:08:04.520 INFO:tasks.rados.rados.0.smithi164.stdout:101: oid 47 updating version 0 to 23 2024-04-26T01:08:04.520 INFO:tasks.rados.rados.0.smithi164.stdout:101: oid 47 version 23 is already newer than 22 2024-04-26T01:08:04.520 INFO:tasks.rados.rados.0.smithi164.stdout:101: oid 47 updating version 23 to 24 2024-04-26T01:08:04.520 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 47 v 24 (ObjNum 65 snap 3 seq_num 65) dirty exists 2024-04-26T01:08:04.520 INFO:tasks.rados.rados.0.smithi164.stdout:101: left oid 47 (ObjNum 65 snap 3 seq_num 65) 2024-04-26T01:08:04.520 INFO:tasks.rados.rados.0.smithi164.stdout:104: finishing copy_from racing read to smithi16428812-27 2024-04-26T01:08:04.520 INFO:tasks.rados.rados.0.smithi164.stdout:104: finishing copy_from to smithi16428812-27 2024-04-26T01:08:04.520 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 27 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-26T01:08:04.520 INFO:tasks.rados.rados.0.smithi164.stdout:102: finishing copy_from racing read to smithi16428812-26 2024-04-26T01:08:04.521 INFO:tasks.rados.rados.0.smithi164.stdout:109: seq_num 67 ranges {573440=753664,1327104=376832} 2024-04-26T01:08:04.521 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:04.516+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:04.529 INFO:tasks.rados.rados.0.smithi164.stdout:109: writing smithi16428812-35 from 573440 to 1327104 tid 1 2024-04-26T01:08:04.532 INFO:tasks.rados.rados.0.smithi164.stdout:109: writing smithi16428812-35 from 1327104 to 1703936 tid 2 2024-04-26T01:08:04.532 INFO:tasks.rados.rados.0.smithi164.stdout:101: done (5 left) 2024-04-26T01:08:04.532 INFO:tasks.rados.rados.0.smithi164.stdout:104: done (4 left) 2024-04-26T01:08:04.532 INFO:tasks.rados.rados.0.smithi164.stdout:110: delete oid 3 current snap is 3 2024-04-26T01:08:04.532 INFO:tasks.rados.rados.0.smithi164.stdout:106: finishing rollback tid 1 to smithi16428812-25 2024-04-26T01:08:04.532 INFO:tasks.rados.rados.0.smithi164.stdout:106: finishing rollback tid 2 to smithi16428812-25 2024-04-26T01:08:04.532 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 25 v 17 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-26T01:08:04.532 INFO:tasks.rados.rados.0.smithi164.stdout:102: finishing copy_from to smithi16428812-26 2024-04-26T01:08:04.532 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 26 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-26T01:08:04.532 INFO:tasks.rados.rados.0.smithi164.stdout:108: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-26T01:08:04.545 INFO:tasks.rados.rados.0.smithi164.stdout:102: done (4 left) 2024-04-26T01:08:04.545 INFO:tasks.rados.rados.0.smithi164.stdout:106: done (3 left) 2024-04-26T01:08:04.545 INFO:tasks.rados.rados.0.smithi164.stdout:108: done (2 left) 2024-04-26T01:08:04.545 INFO:tasks.rados.rados.0.smithi164.stdout:110: done (1 left) 2024-04-26T01:08:04.545 INFO:tasks.rados.rados.0.smithi164.stdout:111: read oid 17 snap -1 2024-04-26T01:08:04.545 INFO:tasks.rados.rados.0.smithi164.stdout:111: expect (ObjNum 60 snap 1 seq_num 60) 2024-04-26T01:08:04.545 INFO:tasks.rados.rados.0.smithi164.stdout:112: snap_create 2024-04-26T01:08:04.545 INFO:tasks.rados.rados.0.smithi164.stdout:109: finishing write tid 1 to smithi16428812-35 2024-04-26T01:08:04.545 INFO:tasks.rados.rados.0.smithi164.stdout:109: finishing write tid 2 to smithi16428812-35 2024-04-26T01:08:04.547 INFO:tasks.rados.rados.0.smithi164.stdout:109: finishing write tid 3 to smithi16428812-35 2024-04-26T01:08:04.551 INFO:tasks.rados.rados.0.smithi164.stdout:109: finishing write tid 4 to smithi16428812-35 2024-04-26T01:08:04.551 INFO:tasks.rados.rados.0.smithi164.stdout:109: oid 35 updating version 0 to 13 2024-04-26T01:08:04.551 INFO:tasks.rados.rados.0.smithi164.stdout:109: oid 35 version 13 is already newer than 12 2024-04-26T01:08:04.551 INFO:tasks.rados.rados.0.smithi164.stdout:109: oid 35 updating version 13 to 14 2024-04-26T01:08:04.551 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 35 v 14 (ObjNum 67 snap 3 seq_num 67) dirty exists 2024-04-26T01:08:04.551 INFO:tasks.rados.rados.0.smithi164.stdout:109: left oid 35 (ObjNum 67 snap 3 seq_num 67) 2024-04-26T01:08:04.602 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:291 3.8s0 3:11b43273:::smithi16428812-17:head [read 0~1294336,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.546009+0000", 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 0.056949039999999999, 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.057005707000000003, 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 291 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.603 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.604 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.604 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.546009+0000", 2024-04-26T01:08:04.604 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.604 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.604 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.604 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.604 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.546009+0000", 2024-04-26T01:08:04.604 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.604 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.604 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.604 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.604 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.546010+0000", 2024-04-26T01:08:04.604 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.8700000000000005e-07 2024-04-26T01:08:04.604 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.604 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.605 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.605 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.546014+0000", 2024-04-26T01:08:04.605 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.5279999999999997e-06 2024-04-26T01:08:04.605 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.605 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.605 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.605 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.546024+0000", 2024-04-26T01:08:04.605 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0008e-05 2024-04-26T01:08:04.605 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.605 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.605 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.605 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.546031+0000", 2024-04-26T01:08:04.606 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.007e-06 2024-04-26T01:08:04.606 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.606 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.606 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.606 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.549981+0000", 2024-04-26T01:08:04.606 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0039497550000000001 2024-04-26T01:08:04.606 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.606 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.606 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:04.606 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.550084+0000", 2024-04-26T01:08:04.606 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000102755 2024-04-26T01:08:04.606 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.606 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.607 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.607 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.607 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-26T01:08:04.607 INFO:teuthology.orchestra.run.smithi012.stdout: "num_ops": 1 2024-04-26T01:08:04.607 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:04.617 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:08:04.620 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:04.617+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:04.721 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:08:04.722 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:04.717+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:04.742 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:04.742 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:04.742 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:08:04.742 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:08:04.742 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:04.756 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:08:04.847 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:04.841+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:04.888 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:04.888 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:04.915 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:04.915 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:04.922 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:04.921+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:04.923 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:04.923 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:08:04.923 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:08:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:08:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:7 3.8s0 3:1b6b0095:::smithi16428812-3:head [append 0~49152,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.302696+0000", 2024-04-26T01:08:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.6177573880000002, 2024-04-26T01:08:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.016152680999999999, 2024-04-26T01:08:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 7 2024-04-26T01:08:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.302696+0000", 2024-04-26T01:08:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.302696+0000", 2024-04-26T01:08:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.302704+0000", 2024-04-26T01:08:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1130000000000001e-06 2024-04-26T01:08:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.926 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.926 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.926 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303173+0000", 2024-04-26T01:08:04.926 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000469109 2024-04-26T01:08:04.926 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.926 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.926 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.926 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303207+0000", 2024-04-26T01:08:04.926 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.3937999999999999e-05 2024-04-26T01:08:04.926 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.926 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.926 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.926 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303229+0000", 2024-04-26T01:08:04.926 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1573000000000002e-05 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304061+0000", 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00083202500000000002 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304318+0000", 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025779299999999998 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.306191+0000", 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0018724010000000001 2024-04-26T01:08:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.928 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.928 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.928 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318520+0000", 2024-04-26T01:08:04.928 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012329028000000001 2024-04-26T01:08:04.928 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.928 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.928 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:04.928 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318628+0000", 2024-04-26T01:08:04.928 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00010778800000000001 2024-04-26T01:08:04.928 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.928 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.928 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.928 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318849+0000", 2024-04-26T01:08:04.928 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022091299999999999 2024-04-26T01:08:04.929 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.929 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.929 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.929 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.929 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.929 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s5 32/30 ECSubWrite(tid=1, reqid=client.4521.0:19, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:04.929 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.303040+0000", 2024-04-26T01:08:04.929 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.6174135410000003, 2024-04-26T01:08:04.929 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0050926979999999997, 2024-04-26T01:08:04.929 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.929 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.929 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.929 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.929 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.929 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.930 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303040+0000", 2024-04-26T01:08:04.930 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.930 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.930 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.930 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.930 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303040+0000", 2024-04-26T01:08:04.930 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.930 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.930 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.930 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.930 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303041+0000", 2024-04-26T01:08:04.930 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.677e-06 2024-04-26T01:08:04.930 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.930 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.930 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.931 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303553+0000", 2024-04-26T01:08:04.931 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00051137700000000003 2024-04-26T01:08:04.931 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.931 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.931 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.931 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303645+0000", 2024-04-26T01:08:04.931 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2300999999999996e-05 2024-04-26T01:08:04.931 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.931 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.931 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.931 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303665+0000", 2024-04-26T01:08:04.931 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0207000000000002e-05 2024-04-26T01:08:04.931 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.931 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.932 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.932 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303824+0000", 2024-04-26T01:08:04.932 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015855200000000001 2024-04-26T01:08:04.932 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.932 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.932 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.932 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303848+0000", 2024-04-26T01:08:04.932 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3876999999999999e-05 2024-04-26T01:08:04.932 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308055+0000", 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0042075690000000004 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308132+0000", 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.7138000000000006e-05 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:8 3.8s0 3:1b6b0095:::smithi16428812-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:04.933 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.303305+0000", 2024-04-26T01:08:04.934 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.6171480970000003, 2024-04-26T01:08:04.934 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.020292667, 2024-04-26T01:08:04.934 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.934 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.934 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:04.934 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:04.934 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:04.934 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:04.934 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 8 2024-04-26T01:08:04.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.934 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.934 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.934 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.934 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303305+0000", 2024-04-26T01:08:04.935 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.935 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.935 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.935 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303305+0000", 2024-04-26T01:08:04.935 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.935 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.935 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.935 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303307+0000", 2024-04-26T01:08:04.935 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7770000000000001e-06 2024-04-26T01:08:04.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.935 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.935 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.935 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303320+0000", 2024-04-26T01:08:04.936 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3086e-05 2024-04-26T01:08:04.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.936 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.936 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.936 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303343+0000", 2024-04-26T01:08:04.936 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3323999999999999e-05 2024-04-26T01:08:04.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.936 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.936 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.936 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303387+0000", 2024-04-26T01:08:04.936 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3741e-05 2024-04-26T01:08:04.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.937 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.937 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.307388+0000", 2024-04-26T01:08:04.937 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0040010430000000001 2024-04-26T01:08:04.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.937 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:04.937 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.307645+0000", 2024-04-26T01:08:04.937 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025681400000000001 2024-04-26T01:08:04.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.937 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.938 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311893+0000", 2024-04-26T01:08:04.938 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.004247626 2024-04-26T01:08:04.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.938 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.938 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.938 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318857+0000", 2024-04-26T01:08:04.938 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0069646760000000004 2024-04-26T01:08:04.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.938 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.938 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:04.938 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323372+0000", 2024-04-26T01:08:04.938 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0045143329999999997 2024-04-26T01:08:04.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.939 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.939 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323598+0000", 2024-04-26T01:08:04.939 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000226247 2024-04-26T01:08:04.939 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.939 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.939 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.939 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:9 3.8s0 3:1b6b0095:::smithi16428812-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:04.939 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.303410+0000", 2024-04-26T01:08:04.939 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.6170430160000002, 2024-04-26T01:08:04.939 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.028697769000000001, 2024-04-26T01:08:04.939 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.940 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.940 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:04.940 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:04.940 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:04.940 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:04.940 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 9 2024-04-26T01:08:04.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.940 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.940 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.940 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.940 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303410+0000", 2024-04-26T01:08:04.941 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.941 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.941 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.941 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303410+0000", 2024-04-26T01:08:04.941 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.941 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.941 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.941 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303412+0000", 2024-04-26T01:08:04.941 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.137e-06 2024-04-26T01:08:04.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.941 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.941 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.941 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303420+0000", 2024-04-26T01:08:04.942 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.3540000000000002e-06 2024-04-26T01:08:04.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.942 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.942 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.942 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303439+0000", 2024-04-26T01:08:04.942 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9227000000000001e-05 2024-04-26T01:08:04.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.942 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.942 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.942 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303455+0000", 2024-04-26T01:08:04.942 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5852999999999999e-05 2024-04-26T01:08:04.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.942 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.942 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.942 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312566+0000", 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0091115209999999992 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312793+0000", 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022636999999999999 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325814+0000", 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013020893 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326042+0000", 2024-04-26T01:08:04.943 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022811899999999999 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.332108+0000", 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0060662950000000002 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 33/30 ECSubWrite(tid=4, reqid=client.4521.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.323649+0000", 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.5968041729999998, 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0067605859999999999, 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.944 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323649+0000", 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323649+0000", 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323651+0000", 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7430000000000001e-06 2024-04-26T01:08:04.945 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.946 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.946 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.946 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323677+0000", 2024-04-26T01:08:04.946 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5711000000000002e-05 2024-04-26T01:08:04.946 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.946 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.946 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.946 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323735+0000", 2024-04-26T01:08:04.946 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.8619999999999998e-05 2024-04-26T01:08:04.946 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.946 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.946 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.946 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323758+0000", 2024-04-26T01:08:04.946 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2351e-05 2024-04-26T01:08:04.946 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.947 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.947 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.947 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323954+0000", 2024-04-26T01:08:04.947 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019601199999999999 2024-04-26T01:08:04.947 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.947 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.947 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.947 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323978+0000", 2024-04-26T01:08:04.947 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4284e-05 2024-04-26T01:08:04.947 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.947 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.947 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.947 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.330333+0000", 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0063547940000000004 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.330410+0000", 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.7070999999999994e-05 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s5 33/30 ECSubWrite(tid=5, reqid=client.4521.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.323685+0000", 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.5967687430000002, 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0064670159999999999, 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.948 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323685+0000", 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323685+0000", 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323685+0000", 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.8999999999999996e-07 2024-04-26T01:08:04.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.950 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.950 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.950 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323700+0000", 2024-04-26T01:08:04.950 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4780000000000001e-05 2024-04-26T01:08:04.950 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.950 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.950 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.950 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324457+0000", 2024-04-26T01:08:04.950 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00075669400000000003 2024-04-26T01:08:04.950 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.950 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.950 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.950 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324478+0000", 2024-04-26T01:08:04.950 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.19e-05 2024-04-26T01:08:04.950 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324956+0000", 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00047727700000000002 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324982+0000", 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5876000000000001e-05 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.330009+0000", 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0050273339999999996 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.951 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.330152+0000", 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014256499999999999 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s5 33/30 ECSubWrite(tid=8, reqid=client.4521.0:37, at_version=33'3, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.324674+0000", 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.5957797300000003, 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0070497340000000002, 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.952 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.953 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.953 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.953 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324674+0000", 2024-04-26T01:08:04.953 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.953 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.953 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.953 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.953 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324674+0000", 2024-04-26T01:08:04.953 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.953 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.953 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.953 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.954 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324675+0000", 2024-04-26T01:08:04.954 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5370000000000001e-06 2024-04-26T01:08:04.954 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.954 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.954 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.954 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324970+0000", 2024-04-26T01:08:04.954 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00029448700000000001 2024-04-26T01:08:04.954 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.954 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.954 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.954 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325122+0000", 2024-04-26T01:08:04.954 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015275600000000001 2024-04-26T01:08:04.954 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.954 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325143+0000", 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0985999999999999e-05 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325543+0000", 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000399486 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325574+0000", 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.1186999999999998e-05 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.955 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.331600+0000", 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0060261999999999998 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.331723+0000", 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012309500000000001 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:49 3.8s0 3:11b43273:::smithi16428812-17:head [append 0~278528,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.342928+0000", 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.577525091, 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0099803930000000006, 2024-04-26T01:08:04.956 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.957 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.957 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:04.957 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:04.957 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:04.957 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:04.957 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 49 2024-04-26T01:08:04.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.957 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.957 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.957 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.957 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.342928+0000", 2024-04-26T01:08:04.957 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.957 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.957 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.958 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.342928+0000", 2024-04-26T01:08:04.958 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.958 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.958 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.958 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.958 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.342931+0000", 2024-04-26T01:08:04.958 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.0000000000000001e-06 2024-04-26T01:08:04.958 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.958 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.958 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.958 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.343938+0000", 2024-04-26T01:08:04.958 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0010073129999999999 2024-04-26T01:08:04.958 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.959 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.959 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.959 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.343964+0000", 2024-04-26T01:08:04.959 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5936999999999999e-05 2024-04-26T01:08:04.959 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.959 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.959 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.959 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.343978+0000", 2024-04-26T01:08:04.959 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3481e-05 2024-04-26T01:08:04.959 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.959 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.959 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.959 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344123+0000", 2024-04-26T01:08:04.959 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014518199999999999 2024-04-26T01:08:04.960 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.960 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.960 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:04.960 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344393+0000", 2024-04-26T01:08:04.960 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00026982599999999997 2024-04-26T01:08:04.960 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.960 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.960 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.960 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.345686+0000", 2024-04-26T01:08:04.960 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001292774 2024-04-26T01:08:04.960 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.960 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.960 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.960 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.351912+0000", 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.006225904 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.352571+0000", 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00065908000000000004 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.352909+0000", 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033789600000000002 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.961 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.962 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:50 3.8s0 3:11b43273:::smithi16428812-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:04.962 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.344014+0000", 2024-04-26T01:08:04.962 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.5764393170000002, 2024-04-26T01:08:04.962 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013017295999999999, 2024-04-26T01:08:04.962 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.962 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.962 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:04.962 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:04.962 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:04.962 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:04.962 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 50 2024-04-26T01:08:04.962 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.962 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.962 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.962 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344014+0000", 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344014+0000", 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344015+0000", 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.437e-06 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.963 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344022+0000", 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.9970000000000001e-06 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344034+0000", 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1817000000000001e-05 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344043+0000", 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.8060000000000001e-06 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.347851+0000", 2024-04-26T01:08:04.964 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0038083570000000001 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.348142+0000", 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00029042400000000001 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350786+0000", 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002644202 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.352937+0000", 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0021514910000000002 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.965 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.356940+0000", 2024-04-26T01:08:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0040022740000000001 2024-04-26T01:08:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357031+0000", 2024-04-26T01:08:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.1490999999999995e-05 2024-04-26T01:08:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:51 3.8s0 3:11b43273:::smithi16428812-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.344056+0000", 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.5763973060000005, 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.01557094, 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 51 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344056+0000", 2024-04-26T01:08:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344056+0000", 2024-04-26T01:08:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344057+0000", 2024-04-26T01:08:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1699999999999997e-07 2024-04-26T01:08:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344060+0000", 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.703e-06 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344071+0000", 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.057e-05 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344085+0000", 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3997e-05 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.351519+0000", 2024-04-26T01:08:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0074343079999999997 2024-04-26T01:08:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.351770+0000", 2024-04-26T01:08:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025095500000000003 2024-04-26T01:08:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357516+0000", 2024-04-26T01:08:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0057452919999999999 2024-04-26T01:08:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357612+0000", 2024-04-26T01:08:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6444e-05 2024-04-26T01:08:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.359627+0000", 2024-04-26T01:08:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002014854 2024-04-26T01:08:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.972 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 33/30 ECSubWrite(tid=17, reqid=client.4521.0:73, at_version=33'3, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:04.972 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.378371+0000", 2024-04-26T01:08:04.972 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.5420818890000003, 2024-04-26T01:08:04.972 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0062186180000000004, 2024-04-26T01:08:04.972 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.972 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.972 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.972 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.972 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.972 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.972 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.378371+0000", 2024-04-26T01:08:04.972 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.972 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.972 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.972 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.973 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.378371+0000", 2024-04-26T01:08:04.973 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.973 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.973 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.973 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.973 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.378373+0000", 2024-04-26T01:08:04.973 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.415e-06 2024-04-26T01:08:04.973 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.973 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.973 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.973 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.378609+0000", 2024-04-26T01:08:04.973 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000235968 2024-04-26T01:08:04.973 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.973 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.974 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.974 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.378677+0000", 2024-04-26T01:08:04.974 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.7759e-05 2024-04-26T01:08:04.974 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.974 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.974 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.974 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.378690+0000", 2024-04-26T01:08:04.974 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.362e-05 2024-04-26T01:08:04.974 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.974 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.974 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.974 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.378786+0000", 2024-04-26T01:08:04.974 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5383999999999993e-05 2024-04-26T01:08:04.974 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.975 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.975 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.975 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.378794+0000", 2024-04-26T01:08:04.975 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.0450000000000001e-06 2024-04-26T01:08:04.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.975 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.975 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.975 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.384528+0000", 2024-04-26T01:08:04.975 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0057341709999999997 2024-04-26T01:08:04.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.975 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.975 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.975 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.384590+0000", 2024-04-26T01:08:04.975 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.2255999999999995e-05 2024-04-26T01:08:04.975 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.976 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.976 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.976 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.976 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.976 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s4 33/30 ECSubWrite(tid=5, reqid=client.4521.0:116, at_version=33'4, trim_to=0'0, roll_forward_to=33'2))", 2024-04-26T01:08:04.976 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.441547+0000", 2024-04-26T01:08:04.976 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.478906469, 2024-04-26T01:08:04.976 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0054677240000000002, 2024-04-26T01:08:04.976 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.976 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.976 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.976 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.976 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.976 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.976 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.441547+0000", 2024-04-26T01:08:04.977 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.977 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.977 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.977 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.977 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.441547+0000", 2024-04-26T01:08:04.977 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.977 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.977 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.977 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.977 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.441548+0000", 2024-04-26T01:08:04.977 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.474e-06 2024-04-26T01:08:04.977 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.977 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.977 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.978 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.441582+0000", 2024-04-26T01:08:04.978 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.3263000000000003e-05 2024-04-26T01:08:04.978 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.978 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.978 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.978 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.441661+0000", 2024-04-26T01:08:04.978 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.9832000000000004e-05 2024-04-26T01:08:04.978 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.978 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.978 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.978 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.441678+0000", 2024-04-26T01:08:04.978 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6426000000000002e-05 2024-04-26T01:08:04.978 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.979 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.979 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.979 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.441798+0000", 2024-04-26T01:08:04.979 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000120345 2024-04-26T01:08:04.979 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.979 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.979 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.979 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.441816+0000", 2024-04-26T01:08:04.979 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7496999999999999e-05 2024-04-26T01:08:04.979 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.979 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.979 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.979 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.446942+0000", 2024-04-26T01:08:04.979 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0051261559999999998 2024-04-26T01:08:04.980 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.980 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.980 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.980 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.447015+0000", 2024-04-26T01:08:04.980 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.2731000000000005e-05 2024-04-26T01:08:04.980 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.980 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.980 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.980 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.980 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.980 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s4 33/30 ECSubWrite(tid=14, reqid=client.4521.0:164, at_version=33'10, trim_to=0'0, roll_forward_to=33'8))", 2024-04-26T01:08:04.980 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.498524+0000", 2024-04-26T01:08:04.980 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.42192922, 2024-04-26T01:08:04.981 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0052345650000000001, 2024-04-26T01:08:04.981 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.981 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.981 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.981 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.981 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.981 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.981 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.498524+0000", 2024-04-26T01:08:04.981 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.981 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.981 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.981 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.981 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.498524+0000", 2024-04-26T01:08:04.981 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.981 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.982 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.982 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.982 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.498525+0000", 2024-04-26T01:08:04.982 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.003e-06 2024-04-26T01:08:04.982 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.982 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.982 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.982 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.498570+0000", 2024-04-26T01:08:04.982 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.4867000000000003e-05 2024-04-26T01:08:04.982 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.983 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.983 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.983 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.498621+0000", 2024-04-26T01:08:04.983 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0905000000000001e-05 2024-04-26T01:08:04.983 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.983 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.983 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.983 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.498634+0000", 2024-04-26T01:08:04.983 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3028e-05 2024-04-26T01:08:04.983 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.983 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.983 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.984 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.498721+0000", 2024-04-26T01:08:04.984 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.6723000000000003e-05 2024-04-26T01:08:04.984 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.984 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.984 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.984 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.498732+0000", 2024-04-26T01:08:04.984 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1216000000000001e-05 2024-04-26T01:08:04.984 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.984 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.984 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.984 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.503674+0000", 2024-04-26T01:08:04.984 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0049424100000000004 2024-04-26T01:08:04.984 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.984 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.984 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.985 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.503759+0000", 2024-04-26T01:08:04.985 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.4413000000000006e-05 2024-04-26T01:08:04.985 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.985 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.985 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.985 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.985 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.985 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s5 33/30 ECSubWrite(tid=49, reqid=client.4521.0:154, at_version=33'11, trim_to=0'0, roll_forward_to=33'10))", 2024-04-26T01:08:04.985 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.500341+0000", 2024-04-26T01:08:04.985 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.4201120569999999, 2024-04-26T01:08:04.985 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0056036059999999997, 2024-04-26T01:08:04.985 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.985 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.985 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.985 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.986 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.986 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.986 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.500341+0000", 2024-04-26T01:08:04.986 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.986 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.986 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.986 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.986 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.500341+0000", 2024-04-26T01:08:04.986 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.986 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.986 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.986 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.986 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.500342+0000", 2024-04-26T01:08:04.986 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2979999999999999e-06 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.500516+0000", 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017366300000000001 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.500628+0000", 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000111475 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.500642+0000", 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4671000000000001e-05 2024-04-26T01:08:04.987 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.500746+0000", 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000104023 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.500763+0000", 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7042e-05 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.505880+0000", 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.005116593 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.988 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.989 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.989 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.505945+0000", 2024-04-26T01:08:04.989 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.4840999999999998e-05 2024-04-26T01:08:04.989 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.989 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.989 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.989 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.989 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.989 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.cs6 38/30 ECSubWrite(tid=73, reqid=client.4521.0:263, at_version=0'0, trim_to=0'0, roll_forward_to=38'22))", 2024-04-26T01:08:04.989 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.413895+0000", 2024-04-26T01:08:04.989 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 0.506558022, 2024-04-26T01:08:04.989 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0056438979999999996, 2024-04-26T01:08:04.989 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.989 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.990 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.990 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.990 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.990 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.990 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.413895+0000", 2024-04-26T01:08:04.990 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.990 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.990 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.990 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.990 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.413895+0000", 2024-04-26T01:08:04.990 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.990 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.990 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.991 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.991 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.413897+0000", 2024-04-26T01:08:04.991 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6199999999999999e-06 2024-04-26T01:08:04.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.991 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.991 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.991 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.413917+0000", 2024-04-26T01:08:04.991 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9746999999999999e-05 2024-04-26T01:08:04.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.991 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.991 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.991 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.413986+0000", 2024-04-26T01:08:04.991 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.9233999999999995e-05 2024-04-26T01:08:04.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.991 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.992 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.992 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.414011+0000", 2024-04-26T01:08:04.992 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4837e-05 2024-04-26T01:08:04.992 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.992 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.992 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.992 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.414082+0000", 2024-04-26T01:08:04.992 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.0915999999999997e-05 2024-04-26T01:08:04.992 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.992 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.992 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.992 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.414089+0000", 2024-04-26T01:08:04.992 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8529999999999992e-06 2024-04-26T01:08:04.992 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.992 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.993 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.993 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.419504+0000", 2024-04-26T01:08:04.993 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0054149000000000003 2024-04-26T01:08:04.993 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.993 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.993 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.993 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.419539+0000", 2024-04-26T01:08:04.993 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.4790999999999999e-05 2024-04-26T01:08:04.993 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.993 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.993 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.993 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.993 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.993 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 38/30 ECSubWrite(tid=85, reqid=osd.4.0:85, at_version=38'19, trim_to=0'0, roll_forward_to=38'15))", 2024-04-26T01:08:04.993 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.432265+0000", 2024-04-26T01:08:04.994 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 0.488188336, 2024-04-26T01:08:04.994 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011946251999999999, 2024-04-26T01:08:04.994 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.994 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.994 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.994 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.994 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.994 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.994 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432265+0000", 2024-04-26T01:08:04.994 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.995 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.995 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.995 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432265+0000", 2024-04-26T01:08:04.995 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.995 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.995 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.995 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432266+0000", 2024-04-26T01:08:04.995 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5900000000000005e-07 2024-04-26T01:08:04.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.995 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.995 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:04.995 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432295+0000", 2024-04-26T01:08:04.995 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9374999999999999e-05 2024-04-26T01:08:04.996 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.996 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.996 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:04.996 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441494+0000", 2024-04-26T01:08:04.996 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0091991369999999996 2024-04-26T01:08:04.996 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.996 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.996 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:04.996 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441529+0000", 2024-04-26T01:08:04.996 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.4677e-05 2024-04-26T01:08:04.996 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.996 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.996 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:04.996 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441702+0000", 2024-04-26T01:08:04.997 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017256500000000001 2024-04-26T01:08:04.997 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.997 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.997 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:04.997 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441731+0000", 2024-04-26T01:08:04.997 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8897000000000001e-05 2024-04-26T01:08:04.997 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.997 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.997 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:04.997 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.444168+0000", 2024-04-26T01:08:04.997 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0024374930000000002 2024-04-26T01:08:04.997 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.997 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.997 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:04.997 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.444211+0000", 2024-04-26T01:08:04.998 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3149000000000003e-05 2024-04-26T01:08:04.998 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.998 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:04.998 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:04.998 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.998 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.998 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 38/30 ECSubWrite(tid=86, reqid=client.4521.0:267, at_version=38'21, trim_to=0'0, roll_forward_to=38'15))", 2024-04-26T01:08:04.998 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.440429+0000", 2024-04-26T01:08:04.998 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 0.48002384100000001, 2024-04-26T01:08:04.998 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0064439479999999997, 2024-04-26T01:08:04.998 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:04.998 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:04.998 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:04.998 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:04.999 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.999 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:04.999 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.440429+0000", 2024-04-26T01:08:04.999 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.999 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.999 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.999 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:04.999 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.440429+0000", 2024-04-26T01:08:04.999 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:04.999 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:04.999 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:04.999 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:04.999 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.440431+0000", 2024-04-26T01:08:04.999 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1930000000000001e-06 2024-04-26T01:08:04.999 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.000 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.000 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:05.000 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441424+0000", 2024-04-26T01:08:05.000 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00099325900000000003 2024-04-26T01:08:05.000 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.000 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.000 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:05.000 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.445195+0000", 2024-04-26T01:08:05.000 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0037713870000000002 2024-04-26T01:08:05.000 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.000 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.000 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:05.001 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.445204+0000", 2024-04-26T01:08:05.001 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.3129999999999999e-06 2024-04-26T01:08:05.001 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.001 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.001 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:05.001 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.445289+0000", 2024-04-26T01:08:05.001 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.5130000000000007e-05 2024-04-26T01:08:05.002 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.002 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.002 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:05.002 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.445297+0000", 2024-04-26T01:08:05.002 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.3629999999999994e-06 2024-04-26T01:08:05.002 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.002 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.003 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:05.003 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.446799+0000", 2024-04-26T01:08:05.003 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001502319 2024-04-26T01:08:05.003 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.003 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.003 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:05.003 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.446873+0000", 2024-04-26T01:08:05.003 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.3984000000000002e-05 2024-04-26T01:08:05.003 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:05.003 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:05.003 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:05.003 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.003 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.003 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:288 3.8s0 3:1b6b0095:::smithi16428812-3:head [delete] snapc 4=[2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.532919+0000", 2024-04-26T01:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 0.38753375200000001, 2024-04-26T01:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0091303889999999992, 2024-04-26T01:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 288 2024-04-26T01:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532919+0000", 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532919+0000", 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532921+0000", 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8810000000000001e-06 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532929+0000", 2024-04-26T01:08:05.005 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8599999999999993e-06 2024-04-26T01:08:05.006 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.006 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.006 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:05.006 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532950+0000", 2024-04-26T01:08:05.006 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0738999999999999e-05 2024-04-26T01:08:05.006 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.006 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.006 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:05.006 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532960+0000", 2024-04-26T01:08:05.006 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5990000000000008e-06 2024-04-26T01:08:05.006 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.006 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.007 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:05.007 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.533062+0000", 2024-04-26T01:08:05.007 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000102402 2024-04-26T01:08:05.007 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.007 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.007 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:05.007 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.533241+0000", 2024-04-26T01:08:05.007 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017890899999999999 2024-04-26T01:08:05.007 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.007 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.007 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:05.007 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.534156+0000", 2024-04-26T01:08:05.007 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00091496199999999996 2024-04-26T01:08:05.007 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.007 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.008 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:05.008 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.535119+0000", 2024-04-26T01:08:05.008 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00096345000000000003 2024-04-26T01:08:05.008 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.008 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.008 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:05.008 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.541980+0000", 2024-04-26T01:08:05.008 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0068602719999999997 2024-04-26T01:08:05.008 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.008 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.008 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:05.008 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.542050+0000", 2024-04-26T01:08:05.008 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.0314999999999993e-05 2024-04-26T01:08:05.008 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:05.009 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:05.009 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:05.009 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.009 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.009 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:289 3.8s0 3:11b43273:::smithi16428812-17:head [read 0~1294336,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:05.009 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.545880+0000", 2024-04-26T01:08:05.009 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 0.374572776, 2024-04-26T01:08:05.009 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0064479150000000002, 2024-04-26T01:08:05.009 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:05.009 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:05.009 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:05.009 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:05.009 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:05.009 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:05.010 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 289 2024-04-26T01:08:05.010 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.010 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:05.010 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.010 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:05.010 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545880+0000", 2024-04-26T01:08:05.010 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:05.010 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.010 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.010 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:05.010 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545880+0000", 2024-04-26T01:08:05.010 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:05.010 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.011 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.011 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:05.011 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545882+0000", 2024-04-26T01:08:05.011 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.934e-06 2024-04-26T01:08:05.011 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.011 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.011 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:05.011 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545891+0000", 2024-04-26T01:08:05.011 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.8580000000000006e-06 2024-04-26T01:08:05.011 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.011 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.011 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:05.011 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545910+0000", 2024-04-26T01:08:05.011 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8702000000000002e-05 2024-04-26T01:08:05.011 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545923+0000", 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2588e-05 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.546011+0000", 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.8372e-05 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.546115+0000", 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000104373 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.012 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.552328+0000", 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0062130880000000003 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:290 3.8s0 3:11b43273:::smithi16428812-17:head [read 0~1294336,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.545945+0000", 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 0.37450834300000002, 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0074298849999999998, 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:05.013 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:05.014 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:05.014 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 290 2024-04-26T01:08:05.014 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.014 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:05.014 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.014 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:05.014 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545945+0000", 2024-04-26T01:08:05.014 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:05.014 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.014 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.014 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:05.014 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545945+0000", 2024-04-26T01:08:05.014 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:05.014 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.014 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545946+0000", 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.0500000000000003e-07 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545951+0000", 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0429999999999998e-06 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545960+0000", 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.3859999999999996e-06 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.015 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:05.016 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545972+0000", 2024-04-26T01:08:05.016 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2234e-05 2024-04-26T01:08:05.016 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.016 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.016 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:05.016 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.547938+0000", 2024-04-26T01:08:05.016 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001965989 2024-04-26T01:08:05.016 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.016 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.016 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:05.016 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.548007+0000", 2024-04-26T01:08:05.016 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.9151000000000003e-05 2024-04-26T01:08:05.016 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:05.016 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:05.016 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:05.017 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.553375+0000", 2024-04-26T01:08:05.017 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0053673770000000004 2024-04-26T01:08:05.017 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:05.017 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:05.017 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:05.017 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:05.017 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:05.017 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:05.018 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:05.018 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:05.019 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:05.019 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:05.019 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:08:05.020 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:05.020 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:05.033 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:05.033 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:05.060 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:05.057+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:05.066 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:05.067 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:05.099 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:05.099 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:05.124 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:05.124+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:05.131 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:05.131 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:05.136 INFO:tasks.rados.rados.0.smithi164.stdout:111: expect (ObjNum 60 snap 1 seq_num 60) 2024-04-26T01:08:05.164 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:05.165 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:05.201 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:05.202 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:05.206 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:08:05.225 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:05.221+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:05.239 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:08:05.240 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:08:05.276 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:05.276 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:05.313 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:05.313 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:05.326 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:05.321+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:05.326 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:05.327 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:05.340 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:08:05.357 INFO:tasks.rados.rados.0.smithi164.stdout:109: done (2 left) 2024-04-26T01:08:05.357 INFO:tasks.rados.rados.0.smithi164.stdout:111: done (1 left) 2024-04-26T01:08:05.357 INFO:tasks.rados.rados.0.smithi164.stdout:112: done (0 left) 2024-04-26T01:08:05.357 INFO:tasks.rados.rados.0.smithi164.stdout:113: read oid 18 snap -1 2024-04-26T01:08:05.357 INFO:tasks.rados.rados.0.smithi164.stdout:113: expect (ObjNum 17 snap 0 seq_num 17) 2024-04-26T01:08:05.357 INFO:tasks.rados.rados.0.smithi164.stdout:114: rmattr oid 9 current snap is 4 2024-04-26T01:08:05.357 INFO:tasks.rados.rados.0.smithi164.stdout:114: done (1 left) 2024-04-26T01:08:05.357 INFO:tasks.rados.rados.0.smithi164.stdout:115: read oid 26 snap -1 2024-04-26T01:08:05.358 INFO:tasks.rados.rados.0.smithi164.stdout:115: expect (ObjNum 49 snap 0 seq_num 49) 2024-04-26T01:08:05.358 INFO:tasks.rados.rados.0.smithi164.stdout:116: rmattr oid 20 current snap is 4 2024-04-26T01:08:05.358 INFO:tasks.rados.rados.0.smithi164.stdout:116: done (2 left) 2024-04-26T01:08:05.358 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 20 current snap is 4 2024-04-26T01:08:05.358 INFO:tasks.rados.rados.0.smithi164.stdout:117: seq_num 69 ranges {327680=786432,1114112=376832} 2024-04-26T01:08:05.358 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:05.359 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:05.359 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:05.357+0000 7f8126d9d640 0 -- 172.21.15.8:0/1604038490 >> [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] conn(0x7f8128226010 msgr2=0x7f8128228400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:08:05.363 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-26T01:08:05.375 INFO:tasks.rados.rados.0.smithi164.stdout:117: writing smithi16428812-20 from 327680 to 1114112 tid 1 2024-04-26T01:08:05.382 INFO:tasks.rados.rados.0.smithi164.stdout:117: writing smithi16428812-20 from 1114112 to 1490944 tid 2 2024-04-26T01:08:05.382 INFO:tasks.rados.rados.0.smithi164.stdout:118: rmattr oid 12 current snap is 4 2024-04-26T01:08:05.382 INFO:tasks.rados.rados.0.smithi164.stdout:118: done (3 left) 2024-04-26T01:08:05.383 INFO:tasks.rados.rados.0.smithi164.stdout:119: delete oid 48 current snap is 4 2024-04-26T01:08:05.383 INFO:tasks.rados.rados.0.smithi164.stdout:115: expect (ObjNum 49 snap 0 seq_num 49) 2024-04-26T01:08:05.416 INFO:tasks.rados.rados.0.smithi164.stdout:113: expect (ObjNum 17 snap 0 seq_num 17) 2024-04-26T01:08:05.427 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:05.428+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:05.427 INFO:tasks.rados.rados.0.smithi164.stdout:117: finishing write tid 1 to smithi16428812-20 2024-04-26T01:08:05.428 INFO:tasks.rados.rados.0.smithi164.stdout:117: finishing write tid 2 to smithi16428812-20 2024-04-26T01:08:05.428 INFO:tasks.rados.rados.0.smithi164.stdout:113: done (3 left) 2024-04-26T01:08:05.428 INFO:tasks.rados.rados.0.smithi164.stdout:115: done (2 left) 2024-04-26T01:08:05.428 INFO:tasks.rados.rados.0.smithi164.stdout:119: done (1 left) 2024-04-26T01:08:05.428 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 16 current snap is 4 2024-04-26T01:08:05.428 INFO:tasks.rados.rados.0.smithi164.stdout:120: seq_num 71 ranges {262144=425984,688128=573440,1261568=196608} 2024-04-26T01:08:05.438 INFO:tasks.rados.rados.0.smithi164.stdout:120: writing smithi16428812-16 from 262144 to 688128 tid 1 2024-04-26T01:08:05.449 INFO:tasks.rados.rados.0.smithi164.stdout:120: writing smithi16428812-16 from 688128 to 1261568 tid 2 2024-04-26T01:08:05.452 INFO:tasks.rados.rados.0.smithi164.stdout:120: writing smithi16428812-16 from 1261568 to 1458176 tid 3 2024-04-26T01:08:05.453 INFO:tasks.rados.rados.0.smithi164.stdout:121: read oid 18 snap -1 2024-04-26T01:08:05.453 INFO:tasks.rados.rados.0.smithi164.stdout:121: expect (ObjNum 17 snap 0 seq_num 17) 2024-04-26T01:08:05.453 INFO:tasks.rados.rados.0.smithi164.stdout:122: copy_from oid 39 from oid 44 current snap is 4 2024-04-26T01:08:05.453 INFO:tasks.rados.rados.0.smithi164.stdout:120: finishing write tid 1 to smithi16428812-16 2024-04-26T01:08:05.453 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 25 current snap is 4 2024-04-26T01:08:05.453 INFO:tasks.rados.rados.0.smithi164.stdout:123: seq_num 72 ranges {409600=442368,851968=720896,1572864=49152} 2024-04-26T01:08:05.463 INFO:tasks.rados.rados.0.smithi164.stdout:123: writing smithi16428812-25 from 409600 to 851968 tid 1 2024-04-26T01:08:05.469 INFO:tasks.rados.rados.0.smithi164.stdout:123: writing smithi16428812-25 from 851968 to 1572864 tid 2 2024-04-26T01:08:05.470 INFO:tasks.rados.rados.0.smithi164.stdout:123: writing smithi16428812-25 from 1572864 to 1622016 tid 3 2024-04-26T01:08:05.470 INFO:tasks.rados.rados.0.smithi164.stdout:124: rollback oid 34 current snap is 4 2024-04-26T01:08:05.470 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 34 to 0 2024-04-26T01:08:05.470 INFO:tasks.rados.rados.0.smithi164.stdout:120: finishing write tid 2 to smithi16428812-16 2024-04-26T01:08:05.470 INFO:tasks.rados.rados.0.smithi164.stdout:121: expect (ObjNum 17 snap 0 seq_num 17) 2024-04-26T01:08:05.471 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:05.471 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:05.482 INFO:tasks.rados.rados.0.smithi164.stdout:120: finishing write tid 3 to smithi16428812-16 2024-04-26T01:08:05.482 INFO:tasks.rados.rados.0.smithi164.stdout:120: finishing write tid 4 to smithi16428812-16 2024-04-26T01:08:05.482 INFO:tasks.rados.rados.0.smithi164.stdout:120: finishing write tid 5 to smithi16428812-16 2024-04-26T01:08:05.482 INFO:tasks.rados.rados.0.smithi164.stdout:120: oid 16 updating version 0 to 15 2024-04-26T01:08:05.482 INFO:tasks.rados.rados.0.smithi164.stdout:120: oid 16 version 15 is already newer than 14 2024-04-26T01:08:05.482 INFO:tasks.rados.rados.0.smithi164.stdout:120: oid 16 version 15 is already newer than 13 2024-04-26T01:08:05.482 INFO:tasks.rados.rados.0.smithi164.stdout:120: oid 16 version 15 is already newer than 12 2024-04-26T01:08:05.482 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 16 v 15 (ObjNum 71 snap 4 seq_num 71) dirty exists 2024-04-26T01:08:05.482 INFO:tasks.rados.rados.0.smithi164.stdout:120: left oid 16 (ObjNum 71 snap 4 seq_num 71) 2024-04-26T01:08:05.483 INFO:tasks.rados.rados.0.smithi164.stdout:120: done (5 left) 2024-04-26T01:08:05.483 INFO:tasks.rados.rados.0.smithi164.stdout:121: done (4 left) 2024-04-26T01:08:05.483 INFO:tasks.rados.rados.0.smithi164.stdout:125: copy_from oid 15 from oid 11 current snap is 4 2024-04-26T01:08:05.483 INFO:tasks.rados.rados.0.smithi164.stdout:123: finishing write tid 1 to smithi16428812-25 2024-04-26T01:08:05.483 INFO:tasks.rados.rados.0.smithi164.stdout:123: finishing write tid 2 to smithi16428812-25 2024-04-26T01:08:05.483 INFO:tasks.rados.rados.0.smithi164.stdout:123: finishing write tid 3 to smithi16428812-25 2024-04-26T01:08:05.483 INFO:tasks.rados.rados.0.smithi164.stdout:126: copy_from oid 8 from oid 2 current snap is 4 2024-04-26T01:08:05.483 INFO:tasks.rados.rados.0.smithi164.stdout:127: snap_remove snap 3 2024-04-26T01:08:05.492 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:08:05.536 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:05.533+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:05.604 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:05.604 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:05.614 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:05.614 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:05.629 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:08:05.629 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:05.628+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:05.729 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:05.728+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:05.799 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:05.799 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:05.813 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:08:05.830 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:05.832+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:05.931 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:05.932+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:05.938 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:05.939 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:05.953 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:08:06.032 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:06.029+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:06.084 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:06.084 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:06.098 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:08:06.133 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:06.129+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:06.215 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:06.215 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:06.229 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:08:06.234 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:06.233+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:06.335 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:06.332+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:06.352 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:06.352 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:06.361 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-04-26T01:08:06.367 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:08:06.369 INFO:tasks.rados.rados.0.smithi164.stdout:127: done (6 left) 2024-04-26T01:08:06.369 INFO:tasks.rados.rados.0.smithi164.stdout:128: delete oid 42 current snap is 4 2024-04-26T01:08:06.370 INFO:tasks.rados.rados.0.smithi164.stdout:123: finishing write tid 4 to smithi16428812-25 2024-04-26T01:08:06.370 INFO:tasks.rados.rados.0.smithi164.stdout:124: finishing rollback tid 0 to smithi16428812-34 2024-04-26T01:08:06.370 INFO:tasks.rados.rados.0.smithi164.stdout:124: finishing rollback tid 1 to smithi16428812-34 2024-04-26T01:08:06.370 INFO:tasks.rados.rados.0.smithi164.stdout:123: finishing write tid 5 to smithi16428812-25 2024-04-26T01:08:06.370 INFO:tasks.rados.rados.0.smithi164.stdout:123: oid 25 updating version 0 to 24 2024-04-26T01:08:06.371 INFO:tasks.rados.rados.0.smithi164.stdout:123: oid 25 updating version 24 to 25 2024-04-26T01:08:06.371 INFO:tasks.rados.rados.0.smithi164.stdout:123: oid 25 version 25 is already newer than 23 2024-04-26T01:08:06.371 INFO:tasks.rados.rados.0.smithi164.stdout:123: oid 25 updating version 25 to 26 2024-04-26T01:08:06.371 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 25 v 26 (ObjNum 72 snap 4 seq_num 72) dirty exists 2024-04-26T01:08:06.371 INFO:tasks.rados.rados.0.smithi164.stdout:123: left oid 25 (ObjNum 72 snap 4 seq_num 72) 2024-04-26T01:08:06.371 INFO:tasks.rados.rados.0.smithi164.stdout:124: finishing rollback tid 2 to smithi16428812-34 2024-04-26T01:08:06.372 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 34 v 30 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-26T01:08:06.372 INFO:tasks.rados.rados.0.smithi164.stdout:125: finishing copy_from to smithi16428812-15 2024-04-26T01:08:06.372 INFO:tasks.rados.rados.0.smithi164.stdout:125: got expected ENOENT (src dne) 2024-04-26T01:08:06.372 INFO:tasks.rados.rados.0.smithi164.stdout:125: finishing copy_from racing read to smithi16428812-15 2024-04-26T01:08:06.372 INFO:tasks.rados.rados.0.smithi164.stdout:126: finishing copy_from racing read to smithi16428812-8 2024-04-26T01:08:06.372 INFO:tasks.rados.rados.0.smithi164.stdout:126: finishing copy_from to smithi16428812-8 2024-04-26T01:08:06.373 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 8 v 10 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-26T01:08:06.373 INFO:tasks.rados.rados.0.smithi164.stdout:117: finishing write tid 3 to smithi16428812-20 2024-04-26T01:08:06.373 INFO:tasks.rados.rados.0.smithi164.stdout:117: finishing write tid 4 to smithi16428812-20 2024-04-26T01:08:06.373 INFO:tasks.rados.rados.0.smithi164.stdout:117: oid 20 updating version 0 to 4 2024-04-26T01:08:06.373 INFO:tasks.rados.rados.0.smithi164.stdout:117: oid 20 updating version 4 to 6 2024-04-26T01:08:06.374 INFO:tasks.rados.rados.0.smithi164.stdout:117: oid 20 version 6 is already newer than 5 2024-04-26T01:08:06.374 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 20 v 6 (ObjNum 69 snap 4 seq_num 69) dirty exists 2024-04-26T01:08:06.374 INFO:tasks.rados.rados.0.smithi164.stdout:117: left oid 20 (ObjNum 69 snap 4 seq_num 69) 2024-04-26T01:08:06.374 INFO:tasks.rados.rados.0.smithi164.stdout:122: finishing copy_from racing read to smithi16428812-39 2024-04-26T01:08:06.374 INFO:tasks.rados.rados.0.smithi164.stdout:122: finishing copy_from to smithi16428812-39 2024-04-26T01:08:06.374 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 39 v 4 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-26T01:08:06.385 INFO:tasks.rados.rados.0.smithi164.stdout:117: done (6 left) 2024-04-26T01:08:06.385 INFO:tasks.rados.rados.0.smithi164.stdout:122: done (5 left) 2024-04-26T01:08:06.385 INFO:tasks.rados.rados.0.smithi164.stdout:123: done (4 left) 2024-04-26T01:08:06.385 INFO:tasks.rados.rados.0.smithi164.stdout:124: done (3 left) 2024-04-26T01:08:06.385 INFO:tasks.rados.rados.0.smithi164.stdout:125: done (2 left) 2024-04-26T01:08:06.385 INFO:tasks.rados.rados.0.smithi164.stdout:126: done (1 left) 2024-04-26T01:08:06.386 INFO:tasks.rados.rados.0.smithi164.stdout:128: done (0 left) 2024-04-26T01:08:06.386 INFO:tasks.rados.rados.0.smithi164.stdout:129: rollback oid 34 current snap is 4 2024-04-26T01:08:06.386 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 34 to 0 2024-04-26T01:08:06.387 INFO:tasks.rados.rados.0.smithi164.stdout:130: delete oid 12 current snap is 4 2024-04-26T01:08:06.395 INFO:tasks.rados.rados.0.smithi164.stdout:129: finishing rollback tid 0 to smithi16428812-34 2024-04-26T01:08:06.397 INFO:tasks.rados.rados.0.smithi164.stdout:129: finishing rollback tid 1 to smithi16428812-34 2024-04-26T01:08:06.401 INFO:tasks.rados.rados.0.smithi164.stdout:129: finishing rollback tid 2 to smithi16428812-34 2024-04-26T01:08:06.401 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 34 v 37 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-26T01:08:06.401 INFO:tasks.rados.rados.0.smithi164.stdout:129: done (1 left) 2024-04-26T01:08:06.402 INFO:tasks.rados.rados.0.smithi164.stdout:130: done (0 left) 2024-04-26T01:08:06.402 INFO:tasks.rados.rados.0.smithi164.stdout:131: copy_from oid 26 from oid 1 current snap is 4 2024-04-26T01:08:06.402 INFO:tasks.rados.rados.0.smithi164.stdout:132: snap_remove snap 0 2024-04-26T01:08:06.438 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:06.437+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:06.529 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:06.529 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:06.536 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:06.536+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:06.544 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:08:06.638 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:06.640+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:06.813 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:06.813 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:06.814 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:08:06.814 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:06.813+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:06.914 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:06.912+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:06.941 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:06.941 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:06.955 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:08:07.015 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:07.013+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:07.116 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:07.116+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:07.121 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:07.121 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:07.137 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:08:07.217 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:07.213+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:07.305 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:07.305 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:07.319 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:08:07.319 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:07.316+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:07.374 INFO:tasks.rados.rados.0.smithi164.stdout:132: done (1 left) 2024-04-26T01:08:07.374 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 38 current snap is 4 2024-04-26T01:08:07.374 INFO:tasks.rados.rados.0.smithi164.stdout:133: seq_num 75 ranges {622592=491520,1114112=770048} 2024-04-26T01:08:07.395 INFO:tasks.rados.rados.0.smithi164.stdout:133: writing smithi16428812-38 from 622592 to 1114112 tid 1 2024-04-26T01:08:07.409 INFO:tasks.rados.rados.0.smithi164.stdout:133: writing smithi16428812-38 from 1114112 to 1884160 tid 2 2024-04-26T01:08:07.411 INFO:tasks.rados.rados.0.smithi164.stdout:134: read oid 25 snap -1 2024-04-26T01:08:07.411 INFO:tasks.rados.rados.0.smithi164.stdout:134: expect (ObjNum 72 snap 4 seq_num 72) 2024-04-26T01:08:07.411 INFO:tasks.rados.rados.0.smithi164.stdout:131: finishing copy_from to smithi16428812-26 2024-04-26T01:08:07.411 INFO:tasks.rados.rados.0.smithi164.stdout:131: got expected ENOENT (src dne) 2024-04-26T01:08:07.411 INFO:tasks.rados.rados.0.smithi164.stdout:131: finishing copy_from racing read to smithi16428812-26 2024-04-26T01:08:07.411 INFO:tasks.rados.rados.0.smithi164.stdout:131: done (2 left) 2024-04-26T01:08:07.411 INFO:tasks.rados.rados.0.smithi164.stdout:135: delete oid 35 current snap is 4 2024-04-26T01:08:07.419 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:07.417+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:07.419 INFO:tasks.rados.rados.0.smithi164.stdout:133: finishing write tid 1 to smithi16428812-38 2024-04-26T01:08:07.424 INFO:tasks.rados.rados.0.smithi164.stdout:133: finishing write tid 2 to smithi16428812-38 2024-04-26T01:08:07.438 INFO:tasks.rados.rados.0.smithi164.stdout:134: expect (ObjNum 72 snap 4 seq_num 72) 2024-04-26T01:08:07.492 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:07.492 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:07.506 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:08:07.519 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:07.517+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:07.595 INFO:tasks.rados.rados.0.smithi164.stdout:134: done (2 left) 2024-04-26T01:08:07.596 INFO:tasks.rados.rados.0.smithi164.stdout:135: done (1 left) 2024-04-26T01:08:07.596 INFO:tasks.rados.rados.0.smithi164.stdout:136: rmattr oid 9 current snap is 4 2024-04-26T01:08:07.596 INFO:tasks.rados.rados.0.smithi164.stdout:136: done (1 left) 2024-04-26T01:08:07.596 INFO:tasks.rados.rados.0.smithi164.stdout:137: copy_from oid 17 from oid 49 current snap is 4 2024-04-26T01:08:07.596 INFO:tasks.rados.rados.0.smithi164.stdout:138: copy_from oid 9 from oid 29 current snap is 4 2024-04-26T01:08:07.596 INFO:tasks.rados.rados.0.smithi164.stdout:139: snap_create 2024-04-26T01:08:07.613 INFO:tasks.rados.rados.0.smithi164.stdout:138: finishing copy_from racing read to smithi16428812-9 2024-04-26T01:08:07.616 INFO:tasks.rados.rados.0.smithi164.stdout:137: finishing copy_from racing read to smithi16428812-17 2024-04-26T01:08:07.620 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:07.620+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:07.623 INFO:tasks.rados.rados.0.smithi164.stdout:137: finishing copy_from to smithi16428812-17 2024-04-26T01:08:07.623 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 17 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-26T01:08:07.628 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:08:07.633 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:07.633 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:07.647 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:08:07.721 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:07.720+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:07.798 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:07.799 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:07.817 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:07.817 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:07.822 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:07.820+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:07.824 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:07.824 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:07.831 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:08:07.853 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:07.853 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:07.883 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:07.884 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:07.915 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:07.915 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:07.923 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:07.920+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:07.948 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:07.948 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:07.958 INFO:tasks.rados.rados.0.smithi164.stdout:133: finishing write tid 3 to smithi16428812-38 2024-04-26T01:08:07.961 INFO:tasks.rados.rados.0.smithi164.stdout:133: finishing write tid 4 to smithi16428812-38 2024-04-26T01:08:07.962 INFO:tasks.rados.rados.0.smithi164.stdout:133: oid 38 updating version 0 to 12 2024-04-26T01:08:07.962 INFO:tasks.rados.rados.0.smithi164.stdout:133: oid 38 updating version 12 to 13 2024-04-26T01:08:07.962 INFO:tasks.rados.rados.0.smithi164.stdout:133: oid 38 version 13 is already newer than 11 2024-04-26T01:08:07.962 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 38 v 13 (ObjNum 75 snap 4 seq_num 75) dirty exists 2024-04-26T01:08:07.962 INFO:tasks.rados.rados.0.smithi164.stdout:133: left oid 38 (ObjNum 75 snap 4 seq_num 75) 2024-04-26T01:08:07.974 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:07.975 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:07.994 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:07.994 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:08.007 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:08.007 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:08.008 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:08:08.024 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:08.020+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:08.037 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:08.037 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:08.070 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:08.070 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:08.105 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:08.105 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:08.125 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:08.124+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:08.140 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:08:08.140 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:08:08.141 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:08.141 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:08.141 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:08:08.141 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:08.154 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:08:08.173 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:08.173 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:08.208 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:08.208 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:08.226 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:08.220+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:08.241 INFO:tasks.rados.rados.0.smithi164.stdout:138: finishing copy_from to smithi16428812-9 2024-04-26T01:08:08.241 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 9 v 10 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-26T01:08:08.246 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:08.246 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:08.246 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:08.244+0000 7f57f5b4b640 0 -- 172.21.15.8:0/1082238071 >> [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] conn(0x7f57f0224040 msgr2=0x7f57f0226430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:08:08.272 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:08.272 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:08.272 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2024-04-26T01:08:08.272 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2024-04-26T01:08:08.272 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:08.286 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:08:08.327 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:08.325+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:08.386 INFO:tasks.rados.rados.0.smithi164.stdout:133: done (3 left) 2024-04-26T01:08:08.386 INFO:tasks.rados.rados.0.smithi164.stdout:137: done (2 left) 2024-04-26T01:08:08.386 INFO:tasks.rados.rados.0.smithi164.stdout:138: done (1 left) 2024-04-26T01:08:08.386 INFO:tasks.rados.rados.0.smithi164.stdout:139: done (0 left) 2024-04-26T01:08:08.386 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 10 current snap is 5 2024-04-26T01:08:08.386 INFO:tasks.rados.rados.0.smithi164.stdout:140: seq_num 77 ranges {163840=524288,688128=524288,1212416=245760} 2024-04-26T01:08:08.398 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-26T01:08:08.398 INFO:tasks.rados.rados.0.smithi164.stdout:140: writing smithi16428812-10 from 163840 to 688128 tid 1 2024-04-26T01:08:08.408 INFO:tasks.rados.rados.0.smithi164.stdout:140: writing smithi16428812-10 from 688128 to 1212416 tid 2 2024-04-26T01:08:08.410 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:08.410 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2024-04-26T01:08:08.410 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2024-04-26T01:08:08.410 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2024-04-26T01:08:08.410 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.410 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.ds1 32/30 ECSubWrite(tid=0, reqid=client.4521.0:4, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:08.410 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.304352+0000", 2024-04-26T01:08:08.410 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.1053758689999995, 2024-04-26T01:08:08.410 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0083134550000000008, 2024-04-26T01:08:08.411 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.411 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.411 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.411 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.411 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.411 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.411 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304352+0000", 2024-04-26T01:08:08.411 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.411 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.411 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.411 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.411 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304352+0000", 2024-04-26T01:08:08.411 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.411 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.412 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.412 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.412 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304353+0000", 2024-04-26T01:08:08.412 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5700000000000002e-07 2024-04-26T01:08:08.412 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.412 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.412 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.412 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304550+0000", 2024-04-26T01:08:08.412 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019731600000000001 2024-04-26T01:08:08.412 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.412 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.412 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.412 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304683+0000", 2024-04-26T01:08:08.412 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000132998 2024-04-26T01:08:08.412 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304726+0000", 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3081999999999998e-05 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304949+0000", 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022243600000000001 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304976+0000", 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7206e-05 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.414 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.414 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.312588+0000", 2024-04-26T01:08:08.414 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0076121670000000004 2024-04-26T01:08:08.414 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.414 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.414 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.414 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.312666+0000", 2024-04-26T01:08:08.414 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7593000000000001e-05 2024-04-26T01:08:08.414 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.414 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.414 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.414 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.414 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.414 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.ds1 32/30 ECSubWrite(tid=1, reqid=client.4521.0:5, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.308139+0000", 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.1015890939999995, 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0045765040000000003, 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308139+0000", 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308139+0000", 2024-04-26T01:08:08.415 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308141+0000", 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.778e-06 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308178+0000", 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6869000000000002e-05 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308277+0000", 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.946e-05 2024-04-26T01:08:08.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308297+0000", 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0157e-05 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308469+0000", 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017191400000000001 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308489+0000", 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9694000000000002e-05 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.418 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.418 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.312673+0000", 2024-04-26T01:08:08.418 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0041840200000000001 2024-04-26T01:08:08.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.418 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.418 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.312716+0000", 2024-04-26T01:08:08.418 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2611999999999998e-05 2024-04-26T01:08:08.418 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.418 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.418 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.418 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 32/30 ECSubWrite(tid=157, reqid=client.4521.0:13, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:08.419 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.310202+0000", 2024-04-26T01:08:08.419 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.0995261569999997, 2024-04-26T01:08:08.419 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0083556310000000005, 2024-04-26T01:08:08.419 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.419 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.419 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.419 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.419 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.419 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.310202+0000", 2024-04-26T01:08:08.419 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.419 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.419 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.310202+0000", 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.310203+0000", 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.257e-06 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.310319+0000", 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000115784 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314884+0000", 2024-04-26T01:08:08.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0045650969999999997 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314909+0000", 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4329e-05 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315108+0000", 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019895399999999999 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315129+0000", 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1662e-05 2024-04-26T01:08:08.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.318477+0000", 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033479120000000002 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.318558+0000", 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.0636000000000005e-05 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 32/30 ECSubWrite(tid=158, reqid=client.4521.0:14, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:08.422 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.314747+0000", 2024-04-26T01:08:08.423 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.0949816870000006, 2024-04-26T01:08:08.423 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.020884725999999999, 2024-04-26T01:08:08.423 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.423 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.423 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.423 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.423 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.423 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314747+0000", 2024-04-26T01:08:08.423 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.423 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.423 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314747+0000", 2024-04-26T01:08:08.423 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314748+0000", 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8700000000000001e-06 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314821+0000", 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2556000000000001e-05 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323944+0000", 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0091227779999999998 2024-04-26T01:08:08.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.425 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.425 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323963+0000", 2024-04-26T01:08:08.425 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9595000000000001e-05 2024-04-26T01:08:08.425 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.425 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.425 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324110+0000", 2024-04-26T01:08:08.425 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014636 2024-04-26T01:08:08.425 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.425 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.425 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324134+0000", 2024-04-26T01:08:08.425 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4453000000000001e-05 2024-04-26T01:08:08.425 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335532+0000", 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011398118 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335631+0000", 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8995999999999999e-05 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.ds1 33/30 ECSubWrite(tid=2, reqid=client.4521.0:5, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:08.426 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.316813+0000", 2024-04-26T01:08:08.427 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.0929153239999998, 2024-04-26T01:08:08.427 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0064126929999999997, 2024-04-26T01:08:08.427 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.427 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.427 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.427 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316813+0000", 2024-04-26T01:08:08.427 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.427 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316813+0000", 2024-04-26T01:08:08.427 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316815+0000", 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6649999999999999e-06 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316842+0000", 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7072000000000001e-05 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316923+0000", 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1705999999999993e-05 2024-04-26T01:08:08.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316942+0000", 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8559e-05 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.317062+0000", 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011990299999999999 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.317081+0000", 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9312000000000001e-05 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.430 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.430 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323159+0000", 2024-04-26T01:08:08.430 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0060774440000000004 2024-04-26T01:08:08.430 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.430 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.430 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.430 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323226+0000", 2024-04-26T01:08:08.430 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7032000000000005e-05 2024-04-26T01:08:08.430 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.430 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.430 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.430 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.430 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.430 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 32/30 ECSubWrite(tid=159, reqid=client.4521.0:31, at_version=32'3, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:08.430 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.323697+0000", 2024-04-26T01:08:08.431 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.0860314029999998, 2024-04-26T01:08:08.431 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013382949, 2024-04-26T01:08:08.431 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.431 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.431 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.431 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.431 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.431 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323697+0000", 2024-04-26T01:08:08.431 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.431 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.431 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.431 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323697+0000", 2024-04-26T01:08:08.431 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323697+0000", 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4000000000000001e-07 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323896+0000", 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019882299999999999 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.328908+0000", 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0050118239999999998 2024-04-26T01:08:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.328926+0000", 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8328000000000001e-05 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329545+0000", 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00061852999999999995 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329608+0000", 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2861999999999995e-05 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336992+0000", 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0073843040000000004 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337080+0000", 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.7638000000000004e-05 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.ds1 33/30 ECSubWrite(tid=4, reqid=client.4521.0:34, at_version=33'3, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.324243+0000", 2024-04-26T01:08:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.0854851189999994, 2024-04-26T01:08:08.435 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011396646, 2024-04-26T01:08:08.435 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.435 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.435 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.435 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.435 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.435 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.435 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324243+0000", 2024-04-26T01:08:08.435 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.435 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.435 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.435 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.435 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324243+0000", 2024-04-26T01:08:08.435 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.435 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324245+0000", 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2280000000000001e-06 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324397+0000", 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015143700000000001 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324490+0000", 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3133000000000004e-05 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324513+0000", 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2864999999999999e-05 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324679+0000", 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000166525 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324694+0000", 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4825e-05 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335561+0000", 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010867185999999999 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335640+0000", 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8447000000000003e-05 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.ds1 33/30 ECSubWrite(tid=5, reqid=client.4521.0:35, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.326781+0000", 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.0829473830000005, 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.008898787, 2024-04-26T01:08:08.438 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326781+0000", 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326781+0000", 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.439 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326783+0000", 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7999999999999999e-06 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326824+0000", 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1421999999999997e-05 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326946+0000", 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012172600000000001 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326968+0000", 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2274000000000001e-05 2024-04-26T01:08:08.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.441 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.441 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.327108+0000", 2024-04-26T01:08:08.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000139703 2024-04-26T01:08:08.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.441 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.441 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.327130+0000", 2024-04-26T01:08:08.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2011e-05 2024-04-26T01:08:08.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.441 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.441 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335646+0000", 2024-04-26T01:08:08.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0085158969999999997 2024-04-26T01:08:08.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.442 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.442 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335680+0000", 2024-04-26T01:08:08.442 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3954000000000001e-05 2024-04-26T01:08:08.442 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.442 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.442 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.442 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/30 ECSubWrite(tid=160, reqid=client.4521.0:32, at_version=33'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:08.442 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.328826+0000", 2024-04-26T01:08:08.442 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.0809022259999992, 2024-04-26T01:08:08.442 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010843787000000001, 2024-04-26T01:08:08.442 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.442 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.443 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.443 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.443 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.443 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.328826+0000", 2024-04-26T01:08:08.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.443 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.443 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.328826+0000", 2024-04-26T01:08:08.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.443 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.443 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.328828+0000", 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7570000000000001e-06 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.328866+0000", 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8260000000000003e-05 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337463+0000", 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0085968439999999993 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337477+0000", 2024-04-26T01:08:08.444 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4620999999999999e-05 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337694+0000", 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00021669500000000001 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337715+0000", 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0503e-05 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339557+0000", 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00184206 2024-04-26T01:08:08.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.446 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.446 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339670+0000", 2024-04-26T01:08:08.446 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000113047 2024-04-26T01:08:08.446 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.446 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.446 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.446 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpRead(3.2s2 33/30 ECSubRead(tid=161, to_read={3:49548768:::smithi16428812-5:head=0,4096,0}, subchunks={3:49548768:::smithi16428812-5:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:08.446 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.337412+0000", 2024-04-26T01:08:08.446 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.0723160759999999, 2024-04-26T01:08:08.446 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0052553979999999997, 2024-04-26T01:08:08.446 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.447 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.447 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.447 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.447 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337412+0000", 2024-04-26T01:08:08.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.447 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337412+0000", 2024-04-26T01:08:08.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.447 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337413+0000", 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2049999999999999e-06 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337432+0000", 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8272000000000001e-05 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341917+0000", 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0044857359999999997 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341951+0000", 2024-04-26T01:08:08.449 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3173000000000002e-05 2024-04-26T01:08:08.449 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.449 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.449 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.449 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.342131+0000", 2024-04-26T01:08:08.449 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000180457 2024-04-26T01:08:08.449 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.449 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.449 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.449 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.342668+0000", 2024-04-26T01:08:08.449 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00053655499999999997 2024-04-26T01:08:08.449 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.449 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.449 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.449 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.450 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpRead(3.2s2 33/30 ECSubRead(tid=163, to_read={3:410d186f:::smithi16428812-11:head=0,4096,0}, subchunks={3:410d186f:::smithi16428812-11:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:08.450 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.342255+0000", 2024-04-26T01:08:08.450 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.0674736130000007, 2024-04-26T01:08:08.450 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.047919758, 2024-04-26T01:08:08.450 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.450 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.450 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.450 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.450 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.450 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.342255+0000", 2024-04-26T01:08:08.450 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.450 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.342255+0000", 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.342256+0000", 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3969999999999999e-06 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.342290+0000", 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4085000000000003e-05 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.452 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.389563+0000", 2024-04-26T01:08:08.452 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.047272886 2024-04-26T01:08:08.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.452 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.389578+0000", 2024-04-26T01:08:08.452 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4678000000000001e-05 2024-04-26T01:08:08.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.452 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.389769+0000", 2024-04-26T01:08:08.452 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019143099999999999 2024-04-26T01:08:08.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.390174+0000", 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00040528100000000001 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/30 ECSubWrite(tid=164, reqid=client.4521.0:79, at_version=33'5, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.389336+0000", 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.0203925940000005, 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0067203230000000003, 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.453 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.454 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.454 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.389336+0000", 2024-04-26T01:08:08.454 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.454 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.454 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.454 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.389336+0000", 2024-04-26T01:08:08.454 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.454 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.454 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.454 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.389341+0000", 2024-04-26T01:08:08.454 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9679999999999997e-06 2024-04-26T01:08:08.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.454 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.389515+0000", 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017436400000000001 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394634+0000", 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.005119513 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394651+0000", 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6603999999999999e-05 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.455 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394757+0000", 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000105531 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394769+0000", 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2223e-05 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.396012+0000", 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012432700000000001 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.396056+0000", 2024-04-26T01:08:08.456 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3850000000000002e-05 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/30 ECSubWrite(tid=165, reqid=client.4521.0:80, at_version=33'6, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.394557+0000", 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.0151712269999997, 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.039358791999999997, 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394557+0000", 2024-04-26T01:08:08.457 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.458 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.458 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.458 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.458 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394557+0000", 2024-04-26T01:08:08.458 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.458 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.458 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.458 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.458 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394559+0000", 2024-04-26T01:08:08.458 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8500000000000001e-06 2024-04-26T01:08:08.458 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.458 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.458 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.458 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394591+0000", 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1967000000000001e-05 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432679+0000", 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.038087835 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432698+0000", 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9459000000000001e-05 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432802+0000", 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000103404 2024-04-26T01:08:08.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432812+0000", 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0969e-05 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.433867+0000", 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010541019999999999 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.433916+0000", 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9206000000000003e-05 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.460 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.461 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.461 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.461 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.461 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.3s6 33/30 ECSubWrite(tid=166, reqid=client.4521.0:109, at_version=33'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:08.461 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.432104+0000", 2024-04-26T01:08:08.461 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 8.9776241120000009, 2024-04-26T01:08:08.461 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042723700000000002, 2024-04-26T01:08:08.461 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.461 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.461 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.461 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.461 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.461 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.461 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432104+0000", 2024-04-26T01:08:08.461 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.462 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.462 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.462 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432104+0000", 2024-04-26T01:08:08.462 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.462 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.462 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.462 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432106+0000", 2024-04-26T01:08:08.462 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1830000000000001e-06 2024-04-26T01:08:08.462 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.462 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.462 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432629+0000", 2024-04-26T01:08:08.462 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00052224499999999996 2024-04-26T01:08:08.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.463 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.463 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434348+0000", 2024-04-26T01:08:08.463 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001719583 2024-04-26T01:08:08.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.463 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.463 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434358+0000", 2024-04-26T01:08:08.463 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6930000000000002e-06 2024-04-26T01:08:08.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.463 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.464 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434474+0000", 2024-04-26T01:08:08.464 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011661400000000001 2024-04-26T01:08:08.464 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.464 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.464 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434507+0000", 2024-04-26T01:08:08.464 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.2957000000000003e-05 2024-04-26T01:08:08.464 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.464 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.464 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.436338+0000", 2024-04-26T01:08:08.464 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0018304300000000001 2024-04-26T01:08:08.464 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.464 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.465 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.436376+0000", 2024-04-26T01:08:08.465 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8665000000000003e-05 2024-04-26T01:08:08.465 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.465 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.465 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.465 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.465 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.465 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.3s6 33/30 ECSubWrite(tid=167, reqid=client.4521.0:110, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:08.465 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.434303+0000", 2024-04-26T01:08:08.465 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 8.9754256320000003, 2024-04-26T01:08:08.465 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0051688970000000004, 2024-04-26T01:08:08.465 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.465 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.465 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.465 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.466 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.466 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.466 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434303+0000", 2024-04-26T01:08:08.466 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.466 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.466 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.466 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434303+0000", 2024-04-26T01:08:08.466 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.466 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.466 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.466 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434303+0000", 2024-04-26T01:08:08.466 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7499999999999999e-07 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434327+0000", 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3431e-05 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.438328+0000", 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0040014600000000001 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.438338+0000", 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7799999999999995e-06 2024-04-26T01:08:08.467 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.438415+0000", 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6911999999999998e-05 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.438428+0000", 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3465000000000001e-05 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.439422+0000", 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00099357200000000007 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.469 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.469 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.439472+0000", 2024-04-26T01:08:08.469 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9502000000000001e-05 2024-04-26T01:08:08.469 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.469 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.469 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.469 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.469 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.469 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.3s6 33/30 ECSubWrite(tid=170, reqid=client.4521.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=33'2))", 2024-04-26T01:08:08.469 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.442037+0000", 2024-04-26T01:08:08.469 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 8.9676911599999993, 2024-04-26T01:08:08.469 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.064321377999999998, 2024-04-26T01:08:08.469 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.469 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.469 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.470 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.470 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.470 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.442037+0000", 2024-04-26T01:08:08.470 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.470 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.470 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.470 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.442037+0000", 2024-04-26T01:08:08.470 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.470 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.470 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.470 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.442038+0000", 2024-04-26T01:08:08.471 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9800000000000002e-07 2024-04-26T01:08:08.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.471 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.471 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.442053+0000", 2024-04-26T01:08:08.471 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4816e-05 2024-04-26T01:08:08.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.471 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.471 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.505728+0000", 2024-04-26T01:08:08.471 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.063674713999999993 2024-04-26T01:08:08.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.471 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.471 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.505736+0000", 2024-04-26T01:08:08.472 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.174e-06 2024-04-26T01:08:08.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.472 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.472 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.505807+0000", 2024-04-26T01:08:08.472 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1203000000000003e-05 2024-04-26T01:08:08.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.472 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.472 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.505816+0000", 2024-04-26T01:08:08.472 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1840000000000005e-06 2024-04-26T01:08:08.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.472 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.472 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.506316+0000", 2024-04-26T01:08:08.473 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00050021600000000005 2024-04-26T01:08:08.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.473 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.473 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.506358+0000", 2024-04-26T01:08:08.473 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2073000000000002e-05 2024-04-26T01:08:08.473 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.473 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.473 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.473 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/30 ECSubWrite(tid=172, reqid=client.4521.0:165, at_version=33'7, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:08:08.473 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.505552+0000", 2024-04-26T01:08:08.474 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 8.9041766570000007, 2024-04-26T01:08:08.474 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0037563710000000001, 2024-04-26T01:08:08.474 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.474 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.474 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.474 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.474 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.474 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.505552+0000", 2024-04-26T01:08:08.474 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.474 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.475 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.505552+0000", 2024-04-26T01:08:08.475 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.475 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.475 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.505553+0000", 2024-04-26T01:08:08.475 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1909999999999999e-06 2024-04-26T01:08:08.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.475 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.475 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.505705+0000", 2024-04-26T01:08:08.475 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015230500000000001 2024-04-26T01:08:08.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.475 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.476 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.508001+0000", 2024-04-26T01:08:08.476 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002296055 2024-04-26T01:08:08.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.476 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.476 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.476 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.508017+0000", 2024-04-26T01:08:08.476 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6256000000000002e-05 2024-04-26T01:08:08.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.476 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.476 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.476 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.508083+0000", 2024-04-26T01:08:08.476 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5835999999999998e-05 2024-04-26T01:08:08.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.476 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.477 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.477 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.508090+0000", 2024-04-26T01:08:08.477 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5560000000000004e-06 2024-04-26T01:08:08.477 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.477 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.477 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.477 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.509190+0000", 2024-04-26T01:08:08.477 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001100455 2024-04-26T01:08:08.477 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.477 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.477 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.477 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.509308+0000", 2024-04-26T01:08:08.477 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000117717 2024-04-26T01:08:08.477 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.478 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.478 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.478 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.478 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.478 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 36/30 ECSubWrite(tid=184, reqid=client.4521.0:225, at_version=36'13, trim_to=0'0, roll_forward_to=33'10))", 2024-04-26T01:08:08.478 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:02.440081+0000", 2024-04-26T01:08:08.478 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 5.9696471500000001, 2024-04-26T01:08:08.478 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0063574779999999997, 2024-04-26T01:08:08.478 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.478 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.478 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.478 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.478 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.478 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.479 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.440081+0000", 2024-04-26T01:08:08.479 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.481 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.482 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.482 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.482 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.440081+0000", 2024-04-26T01:08:08.482 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.482 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.482 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.482 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.482 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.440081+0000", 2024-04-26T01:08:08.482 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9700000000000002e-07 2024-04-26T01:08:08.482 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.482 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.482 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.482 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.440123+0000", 2024-04-26T01:08:08.483 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1319000000000002e-05 2024-04-26T01:08:08.483 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.483 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.483 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.483 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.440215+0000", 2024-04-26T01:08:08.483 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.2264999999999998e-05 2024-04-26T01:08:08.483 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.483 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.483 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.483 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.440238+0000", 2024-04-26T01:08:08.483 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2840000000000002e-05 2024-04-26T01:08:08.483 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.483 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.483 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.483 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.441207+0000", 2024-04-26T01:08:08.484 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00096928499999999996 2024-04-26T01:08:08.484 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.484 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.484 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.484 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.441230+0000", 2024-04-26T01:08:08.484 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3003e-05 2024-04-26T01:08:08.484 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.484 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.484 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.484 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.446356+0000", 2024-04-26T01:08:08.484 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0051255169999999996 2024-04-26T01:08:08.484 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.484 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.484 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.484 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.446439+0000", 2024-04-26T01:08:08.485 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2851999999999999e-05 2024-04-26T01:08:08.485 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.485 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.485 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.485 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.485 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.485 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 36/30 ECSubWrite(tid=185, reqid=client.4521.0:225, at_version=0'0, trim_to=0'0, roll_forward_to=36'13))", 2024-04-26T01:08:08.485 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:02.447653+0000", 2024-04-26T01:08:08.485 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 5.9620755479999996, 2024-04-26T01:08:08.485 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0046212900000000001, 2024-04-26T01:08:08.485 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.485 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.485 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.485 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.486 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.486 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.486 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.447653+0000", 2024-04-26T01:08:08.486 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.486 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.486 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.486 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.486 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.447653+0000", 2024-04-26T01:08:08.486 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.486 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.486 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.486 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.486 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.447655+0000", 2024-04-26T01:08:08.486 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9360000000000002e-06 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.447684+0000", 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9713000000000002e-05 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.447811+0000", 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012631099999999999 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.447837+0000", 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6016999999999998e-05 2024-04-26T01:08:08.487 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.448130+0000", 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00029376000000000001 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.448159+0000", 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8802999999999998e-05 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.452179+0000", 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0040198259999999998 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.488 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.489 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.489 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.452274+0000", 2024-04-26T01:08:08.489 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.4924000000000001e-05 2024-04-26T01:08:08.489 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.489 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.489 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.489 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.489 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.489 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 38/30 ECSubWrite(tid=187, reqid=client.4521.0:272, at_version=38'15, trim_to=0'0, roll_forward_to=36'13))", 2024-04-26T01:08:08.489 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:04.432786+0000", 2024-04-26T01:08:08.489 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 3.9769422350000001, 2024-04-26T01:08:08.489 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0036903259999999998, 2024-04-26T01:08:08.489 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.489 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.489 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.490 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.490 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.490 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.490 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.432786+0000", 2024-04-26T01:08:08.490 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.490 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.490 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.490 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.490 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.432786+0000", 2024-04-26T01:08:08.490 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.490 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.490 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.490 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.490 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.432787+0000", 2024-04-26T01:08:08.491 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7000000000000005e-07 2024-04-26T01:08:08.491 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.491 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.491 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.491 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.433256+0000", 2024-04-26T01:08:08.491 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00046978800000000001 2024-04-26T01:08:08.491 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.491 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.491 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.491 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.433325+0000", 2024-04-26T01:08:08.491 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8771999999999998e-05 2024-04-26T01:08:08.491 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.491 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.491 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.433351+0000", 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5890000000000001e-05 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.433688+0000", 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00033700100000000002 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.433714+0000", 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6285999999999999e-05 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.492 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.436385+0000", 2024-04-26T01:08:08.493 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002670749 2024-04-26T01:08:08.493 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.493 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.493 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.493 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.436476+0000", 2024-04-26T01:08:08.493 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1269999999999999e-05 2024-04-26T01:08:08.493 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.493 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.493 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.493 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.493 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.493 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:08.495 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:08.495 INFO:tasks.rados.rados.0.smithi164.stdout:140: writing smithi16428812-10 from 1212416 to 1458176 tid 3 2024-04-26T01:08:08.495 INFO:tasks.rados.rados.0.smithi164.stdout:141: snap_remove snap 4 2024-04-26T01:08:08.496 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:08:08.497 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:08.497 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:08.489+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:08.595 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:08.593+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:08.624 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:08.624 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:08.625 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:08:08.625 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:08.638 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:08:08.696 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:08.696+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:08.762 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:08.762 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:08.762 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2024-04-26T01:08:08.762 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2024-04-26T01:08:08.763 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:08.776 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:08:08.798 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:08.796+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:08.898 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:08.896+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:08.899 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:08.899 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2024-04-26T01:08:08.900 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2024-04-26T01:08:08.900 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2024-04-26T01:08:08.900 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.900 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:4 3.ds0 3:bcd9cb81:::smithi16428812-2:head [append 0~32768,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:08.900 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.302987+0000", 2024-04-26T01:08:08.900 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5972129039999992, 2024-04-26T01:08:08.900 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012685460000000001, 2024-04-26T01:08:08.900 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.900 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.900 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:08.900 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:08.900 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:08.900 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:08.900 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 4 2024-04-26T01:08:08.901 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.901 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.901 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.901 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.901 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.302987+0000", 2024-04-26T01:08:08.901 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.901 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.901 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.901 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.901 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.302987+0000", 2024-04-26T01:08:08.901 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.901 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.901 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.901 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.902 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.302989+0000", 2024-04-26T01:08:08.902 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.266e-06 2024-04-26T01:08:08.902 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.902 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.902 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.902 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303040+0000", 2024-04-26T01:08:08.902 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1029000000000002e-05 2024-04-26T01:08:08.902 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.902 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.902 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.902 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303059+0000", 2024-04-26T01:08:08.902 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8909000000000001e-05 2024-04-26T01:08:08.902 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.903 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.903 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.903 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303074+0000", 2024-04-26T01:08:08.903 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4812e-05 2024-04-26T01:08:08.903 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.903 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.903 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.903 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303197+0000", 2024-04-26T01:08:08.903 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000122748 2024-04-26T01:08:08.903 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.903 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.903 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:08.903 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303424+0000", 2024-04-26T01:08:08.903 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022748500000000001 2024-04-26T01:08:08.904 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.904 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.904 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.904 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304556+0000", 2024-04-26T01:08:08.904 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001131948 2024-04-26T01:08:08.904 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.904 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.904 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.904 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.313947+0000", 2024-04-26T01:08:08.904 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0093914069999999992 2024-04-26T01:08:08.904 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.904 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.904 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:08.904 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315465+0000", 2024-04-26T01:08:08.905 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001517673 2024-04-26T01:08:08.905 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.905 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.905 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.905 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315672+0000", 2024-04-26T01:08:08.905 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000207183 2024-04-26T01:08:08.905 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.905 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.905 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.905 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.905 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.905 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:5 3.ds0 3:bcd9cb81:::smithi16428812-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:08.905 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.303108+0000", 2024-04-26T01:08:08.905 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5970919059999993, 2024-04-26T01:08:08.905 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013204789, 2024-04-26T01:08:08.906 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.906 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.906 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:08.906 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:08.906 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:08.906 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:08.906 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 5 2024-04-26T01:08:08.906 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.906 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.906 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.906 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.906 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303108+0000", 2024-04-26T01:08:08.906 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.906 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303108+0000", 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303109+0000", 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3149999999999999e-06 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303117+0000", 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1899999999999995e-06 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.907 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.908 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.908 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303131+0000", 2024-04-26T01:08:08.908 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3890000000000001e-05 2024-04-26T01:08:08.908 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.908 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.908 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.908 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303143+0000", 2024-04-26T01:08:08.908 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1773999999999999e-05 2024-04-26T01:08:08.908 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.908 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.908 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.305710+0000", 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0025668930000000002 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.305934+0000", 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022386800000000001 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.307007+0000", 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010735930000000001 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.909 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.313953+0000", 2024-04-26T01:08:08.910 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006945657 2024-04-26T01:08:08.910 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.910 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.910 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:08.910 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316089+0000", 2024-04-26T01:08:08.910 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021360509999999999 2024-04-26T01:08:08.910 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.910 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.910 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.910 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316313+0000", 2024-04-26T01:08:08.910 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000223558 2024-04-26T01:08:08.910 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.910 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.911 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.911 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.911 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.911 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:6 3.ds0 3:bcd9cb81:::smithi16428812-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:08.911 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.303162+0000", 2024-04-26T01:08:08.911 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5970379240000003, 2024-04-26T01:08:08.911 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.027157324, 2024-04-26T01:08:08.911 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.911 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.911 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:08.911 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:08.911 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:08.911 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:08.911 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 6 2024-04-26T01:08:08.911 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.912 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.912 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.912 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.912 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303162+0000", 2024-04-26T01:08:08.912 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.912 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.912 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.912 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303162+0000", 2024-04-26T01:08:08.912 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.912 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.912 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.912 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303163+0000", 2024-04-26T01:08:08.912 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.145e-06 2024-04-26T01:08:08.913 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.913 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.913 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.913 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303169+0000", 2024-04-26T01:08:08.913 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8150000000000002e-06 2024-04-26T01:08:08.913 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.913 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.913 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.913 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303184+0000", 2024-04-26T01:08:08.913 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5116000000000001e-05 2024-04-26T01:08:08.913 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.913 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.913 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.913 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303196+0000", 2024-04-26T01:08:08.913 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1778e-05 2024-04-26T01:08:08.914 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.914 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.914 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.914 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308497+0000", 2024-04-26T01:08:08.914 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0053009449999999996 2024-04-26T01:08:08.914 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.914 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.914 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:08.914 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308937+0000", 2024-04-26T01:08:08.914 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00044013100000000002 2024-04-26T01:08:08.914 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.914 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.317636+0000", 2024-04-26T01:08:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0086997570000000007 2024-04-26T01:08:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.317809+0000", 2024-04-26T01:08:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017204899999999999 2024-04-26T01:08:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.330319+0000", 2024-04-26T01:08:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012510588 2024-04-26T01:08:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.916 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.916 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.916 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.916 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.916 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 32/30 ECSubWriteReply(tid=1, last_complete=32'2, committed=1, applied=1))", 2024-04-26T01:08:08.916 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.308880+0000", 2024-04-26T01:08:08.916 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5913193959999994, 2024-04-26T01:08:08.916 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.005358041, 2024-04-26T01:08:08.916 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.916 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.916 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.916 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.916 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.917 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.917 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308880+0000", 2024-04-26T01:08:08.917 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.917 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.917 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.917 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.917 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308880+0000", 2024-04-26T01:08:08.917 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.917 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.917 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.917 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.917 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308882+0000", 2024-04-26T01:08:08.917 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9139999999999998e-06 2024-04-26T01:08:08.917 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.917 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.918 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.918 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308891+0000", 2024-04-26T01:08:08.918 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1279999999999997e-06 2024-04-26T01:08:08.918 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.918 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.918 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.918 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308999+0000", 2024-04-26T01:08:08.918 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010787 2024-04-26T01:08:08.918 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.918 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.918 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.918 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309015+0000", 2024-04-26T01:08:08.918 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5732000000000001e-05 2024-04-26T01:08:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314175+0000", 2024-04-26T01:08:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0051600719999999999 2024-04-26T01:08:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314238+0000", 2024-04-26T01:08:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3324999999999994e-05 2024-04-26T01:08:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 32/30 ECSubWriteReply(tid=0, last_complete=32'1, committed=1, applied=1))", 2024-04-26T01:08:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.309144+0000", 2024-04-26T01:08:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5910558350000006, 2024-04-26T01:08:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0053185649999999999, 2024-04-26T01:08:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309144+0000", 2024-04-26T01:08:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309144+0000", 2024-04-26T01:08:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309145+0000", 2024-04-26T01:08:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.328e-06 2024-04-26T01:08:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309158+0000", 2024-04-26T01:08:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2867000000000001e-05 2024-04-26T01:08:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309240+0000", 2024-04-26T01:08:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1558999999999996e-05 2024-04-26T01:08:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309254+0000", 2024-04-26T01:08:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4423999999999999e-05 2024-04-26T01:08:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314413+0000", 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0051590550000000001 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314462+0000", 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9332000000000001e-05 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 33/30 ECSubWriteReply(tid=1, last_complete=32'2, committed=1, applied=1))", 2024-04-26T01:08:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.309264+0000", 2024-04-26T01:08:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5909357069999999, 2024-04-26T01:08:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0055114789999999997, 2024-04-26T01:08:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309264+0000", 2024-04-26T01:08:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309264+0000", 2024-04-26T01:08:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309266+0000", 2024-04-26T01:08:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8369999999999999e-06 2024-04-26T01:08:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309277+0000", 2024-04-26T01:08:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0991e-05 2024-04-26T01:08:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309352+0000", 2024-04-26T01:08:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4901999999999995e-05 2024-04-26T01:08:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309371+0000", 2024-04-26T01:08:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8828e-05 2024-04-26T01:08:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314745+0000", 2024-04-26T01:08:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0053747309999999998 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314775+0000", 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0190000000000001e-05 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 32/30 ECSubWriteReply(tid=1, last_complete=32'2, committed=1, applied=1))", 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.309299+0000", 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5909003879999997, 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056645660000000002, 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309299+0000", 2024-04-26T01:08:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309299+0000", 2024-04-26T01:08:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309300+0000", 2024-04-26T01:08:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0300000000000001e-06 2024-04-26T01:08:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309308+0000", 2024-04-26T01:08:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9629999999999998e-06 2024-04-26T01:08:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309369+0000", 2024-04-26T01:08:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0223000000000003e-05 2024-04-26T01:08:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309383+0000", 2024-04-26T01:08:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4728e-05 2024-04-26T01:08:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314927+0000", 2024-04-26T01:08:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0055432650000000003 2024-04-26T01:08:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314964+0000", 2024-04-26T01:08:08.931 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7357000000000002e-05 2024-04-26T01:08:08.931 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.931 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.931 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.931 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.931 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.931 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 33/30 ECSubWriteReply(tid=1, last_complete=32'2, committed=1, applied=1))", 2024-04-26T01:08:08.931 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.310885+0000", 2024-04-26T01:08:08.931 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5893143290000005, 2024-04-26T01:08:08.931 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0048923259999999998, 2024-04-26T01:08:08.931 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.931 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.932 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.932 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.932 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.932 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.932 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.310885+0000", 2024-04-26T01:08:08.932 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.932 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.932 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.932 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.932 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.310885+0000", 2024-04-26T01:08:08.932 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.932 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.932 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.933 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.933 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.310887+0000", 2024-04-26T01:08:08.933 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6500000000000001e-06 2024-04-26T01:08:08.933 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.933 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.933 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.933 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.310908+0000", 2024-04-26T01:08:08.933 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0727e-05 2024-04-26T01:08:08.933 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.933 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.933 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.933 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.311047+0000", 2024-04-26T01:08:08.933 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000139667 2024-04-26T01:08:08.934 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.934 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.934 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.934 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.311065+0000", 2024-04-26T01:08:08.934 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7368e-05 2024-04-26T01:08:08.934 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.934 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.934 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.934 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315743+0000", 2024-04-26T01:08:08.934 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0046785419999999999 2024-04-26T01:08:08.934 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.934 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.935 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.935 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315778+0000", 2024-04-26T01:08:08.935 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4372000000000002e-05 2024-04-26T01:08:08.935 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.935 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.935 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.935 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.935 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.935 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:34 3.ds0 3:b1366e28:::smithi16428812-12:head [append 0~196608,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:08.935 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321139+0000", 2024-04-26T01:08:08.935 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.579060686, 2024-04-26T01:08:08.935 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015400954999999999, 2024-04-26T01:08:08.935 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.936 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.936 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:08.936 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:08.936 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:08.936 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:08.936 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 34 2024-04-26T01:08:08.936 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.936 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.936 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.936 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.936 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321139+0000", 2024-04-26T01:08:08.936 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.936 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.937 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.937 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.937 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321139+0000", 2024-04-26T01:08:08.937 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.937 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.937 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.937 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.937 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321142+0000", 2024-04-26T01:08:08.937 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8660000000000002e-06 2024-04-26T01:08:08.937 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.937 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.937 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.938 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321475+0000", 2024-04-26T01:08:08.938 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00033334599999999999 2024-04-26T01:08:08.938 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.938 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.938 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.938 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321504+0000", 2024-04-26T01:08:08.938 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8445e-05 2024-04-26T01:08:08.938 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.938 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.938 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.938 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321518+0000", 2024-04-26T01:08:08.938 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4708e-05 2024-04-26T01:08:08.938 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322532+0000", 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001013246 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322688+0000", 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000156525 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323904+0000", 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012160160000000001 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.939 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.940 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.940 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.332759+0000", 2024-04-26T01:08:08.940 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0088551199999999993 2024-04-26T01:08:08.940 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.940 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.940 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:08.940 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336395+0000", 2024-04-26T01:08:08.940 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0036352559999999999 2024-04-26T01:08:08.940 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.940 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.940 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.940 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336540+0000", 2024-04-26T01:08:08.940 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000145427 2024-04-26T01:08:08.940 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.941 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.941 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.941 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.941 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.941 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:35 3.ds0 3:b1366e28:::smithi16428812-12:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:08.941 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321544+0000", 2024-04-26T01:08:08.941 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5786555680000003, 2024-04-26T01:08:08.941 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015414796, 2024-04-26T01:08:08.941 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.941 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.941 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:08.941 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:08.941 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:08.941 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:08.942 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 35 2024-04-26T01:08:08.942 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.942 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.942 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.942 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.942 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321544+0000", 2024-04-26T01:08:08.942 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.942 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.942 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.942 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.942 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321544+0000", 2024-04-26T01:08:08.942 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.942 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.943 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.943 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.943 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321545+0000", 2024-04-26T01:08:08.943 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3340000000000001e-06 2024-04-26T01:08:08.943 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.943 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.943 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.943 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321553+0000", 2024-04-26T01:08:08.943 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4739999999999999e-06 2024-04-26T01:08:08.943 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.943 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.943 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.943 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321571+0000", 2024-04-26T01:08:08.944 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7688000000000001e-05 2024-04-26T01:08:08.944 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.944 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.944 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.944 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321614+0000", 2024-04-26T01:08:08.944 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3047000000000003e-05 2024-04-26T01:08:08.944 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.944 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.944 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.944 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324961+0000", 2024-04-26T01:08:08.944 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003347184 2024-04-26T01:08:08.944 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.944 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.944 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:08.944 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.325113+0000", 2024-04-26T01:08:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000152287 2024-04-26T01:08:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.327055+0000", 2024-04-26T01:08:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001942292 2024-04-26T01:08:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.332763+0000", 2024-04-26T01:08:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0057070819999999996 2024-04-26T01:08:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336824+0000", 2024-04-26T01:08:08.946 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0040615449999999997 2024-04-26T01:08:08.946 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.946 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.946 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.946 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336959+0000", 2024-04-26T01:08:08.946 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013486300000000001 2024-04-26T01:08:08.946 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.946 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.946 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.946 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.946 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.946 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:36 3.ds0 3:b1366e28:::smithi16428812-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:08.946 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321633+0000", 2024-04-26T01:08:08.947 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5785667310000004, 2024-04-26T01:08:08.947 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.019982366000000001, 2024-04-26T01:08:08.947 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.947 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.947 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:08.947 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:08.947 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:08.947 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:08.947 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 36 2024-04-26T01:08:08.947 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.947 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.948 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.948 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.948 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321633+0000", 2024-04-26T01:08:08.948 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.948 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.948 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.948 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.948 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321633+0000", 2024-04-26T01:08:08.948 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.948 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.948 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.948 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.948 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321634+0000", 2024-04-26T01:08:08.948 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.268e-06 2024-04-26T01:08:08.948 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321641+0000", 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5100000000000004e-06 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321656+0000", 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5191000000000001e-05 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321666+0000", 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0178e-05 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.949 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.950 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.950 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.333161+0000", 2024-04-26T01:08:08.950 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011494605999999999 2024-04-26T01:08:08.950 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.950 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.950 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:08.950 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.333311+0000", 2024-04-26T01:08:08.950 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015046100000000001 2024-04-26T01:08:08.950 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.950 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.950 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.951 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338017+0000", 2024-04-26T01:08:08.951 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0047055489999999998 2024-04-26T01:08:08.951 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.951 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.951 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:08.951 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338172+0000", 2024-04-26T01:08:08.951 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015476500000000001 2024-04-26T01:08:08.951 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.951 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.951 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.951 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341615+0000", 2024-04-26T01:08:08.951 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0034438379999999998 2024-04-26T01:08:08.951 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.952 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.952 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.952 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.952 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.952 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpReadReply(3.ds0 33/30 ECSubReadReply(tid=3, attrs_read=0))", 2024-04-26T01:08:08.952 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.322592+0000", 2024-04-26T01:08:08.952 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5776077199999996, 2024-04-26T01:08:08.952 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077557909999999997, 2024-04-26T01:08:08.952 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.952 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.952 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.952 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.953 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.953 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.953 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322592+0000", 2024-04-26T01:08:08.953 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.953 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.953 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.953 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.953 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322592+0000", 2024-04-26T01:08:08.953 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.953 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.953 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.953 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.953 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322594+0000", 2024-04-26T01:08:08.954 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7740000000000001e-06 2024-04-26T01:08:08.954 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.954 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.954 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.954 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322617+0000", 2024-04-26T01:08:08.954 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3074999999999998e-05 2024-04-26T01:08:08.954 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.954 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.954 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.954 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322695+0000", 2024-04-26T01:08:08.954 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7892999999999994e-05 2024-04-26T01:08:08.954 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.954 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.954 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322710+0000", 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.505e-05 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329473+0000", 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006763682 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.330348+0000", 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000874317 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.956 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.956 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 32/30 ECSubWriteReply(tid=2, last_complete=32'2, committed=1, applied=1))", 2024-04-26T01:08:08.956 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.323325+0000", 2024-04-26T01:08:08.956 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5768743829999998, 2024-04-26T01:08:08.956 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0060949649999999999, 2024-04-26T01:08:08.956 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.956 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.956 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.956 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.956 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.956 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.956 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323325+0000", 2024-04-26T01:08:08.956 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.957 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.957 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.957 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.957 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323325+0000", 2024-04-26T01:08:08.957 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.957 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.957 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.957 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.957 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323327+0000", 2024-04-26T01:08:08.957 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.195e-06 2024-04-26T01:08:08.957 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.957 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.957 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.957 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323334+0000", 2024-04-26T01:08:08.958 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2130000000000002e-06 2024-04-26T01:08:08.958 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.958 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.958 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.958 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323418+0000", 2024-04-26T01:08:08.958 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4499999999999994e-05 2024-04-26T01:08:08.958 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.958 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.958 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.958 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323434+0000", 2024-04-26T01:08:08.958 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5338000000000001e-05 2024-04-26T01:08:08.958 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.958 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.958 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.959 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329342+0000", 2024-04-26T01:08:08.959 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0059083399999999998 2024-04-26T01:08:08.959 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.959 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.959 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.959 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329420+0000", 2024-04-26T01:08:08.959 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8379000000000003e-05 2024-04-26T01:08:08.959 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.959 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.959 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.959 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.959 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.959 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 32/30 ECSubWrite(tid=0, reqid=client.4521.0:28, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:08.960 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.326088+0000", 2024-04-26T01:08:08.960 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5741118850000007, 2024-04-26T01:08:08.960 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0097951900000000005, 2024-04-26T01:08:08.960 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.960 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.960 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.960 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.960 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.960 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.960 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326088+0000", 2024-04-26T01:08:08.960 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.960 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.960 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.960 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.961 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326088+0000", 2024-04-26T01:08:08.961 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.961 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.961 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.961 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.961 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326089+0000", 2024-04-26T01:08:08.961 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5460000000000001e-06 2024-04-26T01:08:08.961 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.961 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.961 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.961 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326154+0000", 2024-04-26T01:08:08.961 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4979000000000002e-05 2024-04-26T01:08:08.961 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.962 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.962 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.962 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326251+0000", 2024-04-26T01:08:08.962 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7139999999999995e-05 2024-04-26T01:08:08.962 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.962 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.962 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.962 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326266+0000", 2024-04-26T01:08:08.962 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4355999999999999e-05 2024-04-26T01:08:08.962 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.962 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.962 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.962 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.330888+0000", 2024-04-26T01:08:08.963 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0046226310000000003 2024-04-26T01:08:08.963 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.963 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.963 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.963 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.331237+0000", 2024-04-26T01:08:08.963 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000348103 2024-04-26T01:08:08.963 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.963 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.963 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.963 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335801+0000", 2024-04-26T01:08:08.963 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0045642189999999996 2024-04-26T01:08:08.963 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.963 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.963 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.964 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335883+0000", 2024-04-26T01:08:08.964 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2216e-05 2024-04-26T01:08:08.964 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.964 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.964 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.964 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.964 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.964 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:46 3.ds0 3:b85e6c81:::smithi16428812-16:head [append 0~262144,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:08.964 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.338039+0000", 2024-04-26T01:08:08.964 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5621607970000007, 2024-04-26T01:08:08.964 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0092704039999999995, 2024-04-26T01:08:08.964 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.965 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.965 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:08.965 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:08.965 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:08.965 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:08.965 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 46 2024-04-26T01:08:08.965 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.965 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.965 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.965 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.965 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338039+0000", 2024-04-26T01:08:08.965 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.965 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.966 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.966 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.966 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338039+0000", 2024-04-26T01:08:08.966 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.966 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.966 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.966 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.966 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338041+0000", 2024-04-26T01:08:08.966 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4660000000000002e-06 2024-04-26T01:08:08.966 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.966 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.966 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.966 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338148+0000", 2024-04-26T01:08:08.967 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000106384 2024-04-26T01:08:08.967 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.967 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.967 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.967 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338170+0000", 2024-04-26T01:08:08.967 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2022e-05 2024-04-26T01:08:08.967 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.967 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.967 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.967 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338184+0000", 2024-04-26T01:08:08.967 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4345000000000001e-05 2024-04-26T01:08:08.967 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.967 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.967 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.968 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341687+0000", 2024-04-26T01:08:08.968 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003502631 2024-04-26T01:08:08.968 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.968 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.968 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:08.968 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341929+0000", 2024-04-26T01:08:08.968 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024248 2024-04-26T01:08:08.968 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.968 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.968 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.968 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.343657+0000", 2024-04-26T01:08:08.968 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001727302 2024-04-26T01:08:08.968 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.968 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.968 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.969 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346591+0000", 2024-04-26T01:08:08.969 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0029344420000000002 2024-04-26T01:08:08.969 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.969 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.969 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:08.969 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.347148+0000", 2024-04-26T01:08:08.969 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000556801 2024-04-26T01:08:08.969 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.969 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.969 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.969 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.347309+0000", 2024-04-26T01:08:08.969 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000161531 2024-04-26T01:08:08.969 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.969 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.970 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.970 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.970 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.970 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:47 3.ds0 3:b85e6c81:::smithi16428812-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:08.970 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.338373+0000", 2024-04-26T01:08:08.970 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5618271149999998, 2024-04-26T01:08:08.970 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012969223, 2024-04-26T01:08:08.970 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.970 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.970 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:08.970 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:08.970 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:08.970 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:08.970 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 47 2024-04-26T01:08:08.971 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.971 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.971 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.971 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.971 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338373+0000", 2024-04-26T01:08:08.971 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.971 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.971 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.971 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.971 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338373+0000", 2024-04-26T01:08:08.971 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.971 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.972 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.972 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.972 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338374+0000", 2024-04-26T01:08:08.972 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.782e-06 2024-04-26T01:08:08.972 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.972 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.972 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.972 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338383+0000", 2024-04-26T01:08:08.972 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.6570000000000003e-06 2024-04-26T01:08:08.972 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.972 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.972 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.972 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338402+0000", 2024-04-26T01:08:08.972 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8708000000000001e-05 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338415+0000", 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2976000000000001e-05 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.344766+0000", 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0063516110000000001 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.344969+0000", 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00020233 2024-04-26T01:08:08.973 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.974 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.974 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.974 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.345881+0000", 2024-04-26T01:08:08.974 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00091244999999999998 2024-04-26T01:08:08.974 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.974 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.974 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.974 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.347458+0000", 2024-04-26T01:08:08.974 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001576945 2024-04-26T01:08:08.974 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.974 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.974 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:08.974 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.351176+0000", 2024-04-26T01:08:08.974 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0037184449999999999 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.351342+0000", 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016531900000000001 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:48 3.ds0 3:b85e6c81:::smithi16428812-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.338562+0000", 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5616375849999997, 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.028604862000000002, 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.975 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.976 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:08.976 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:08.976 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:08.976 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:08.976 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 48 2024-04-26T01:08:08.976 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.976 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.976 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.976 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.976 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338562+0000", 2024-04-26T01:08:08.976 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.976 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.976 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.976 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338562+0000", 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338564+0000", 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5319999999999999e-06 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338572+0000", 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1780000000000003e-06 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.977 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338587+0000", 2024-04-26T01:08:08.978 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5293000000000001e-05 2024-04-26T01:08:08.978 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.978 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.978 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.978 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338599+0000", 2024-04-26T01:08:08.978 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1968e-05 2024-04-26T01:08:08.978 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.978 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.978 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.978 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346313+0000", 2024-04-26T01:08:08.978 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077139060000000004 2024-04-26T01:08:08.978 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.978 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.979 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:08.979 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346451+0000", 2024-04-26T01:08:08.979 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013819400000000001 2024-04-26T01:08:08.979 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.979 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.979 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.979 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.352103+0000", 2024-04-26T01:08:08.979 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056514249999999999 2024-04-26T01:08:08.979 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.979 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.979 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:08.979 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.352448+0000", 2024-04-26T01:08:08.979 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034553299999999997 2024-04-26T01:08:08.979 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.980 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.980 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.980 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.367167+0000", 2024-04-26T01:08:08.980 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.014718833000000001 2024-04-26T01:08:08.980 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.980 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.980 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.980 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.980 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.980 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/30 ECSubWrite(tid=2, reqid=client.4521.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:08.980 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.339308+0000", 2024-04-26T01:08:08.980 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.5608914419999991, 2024-04-26T01:08:08.980 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0072084560000000002, 2024-04-26T01:08:08.980 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.980 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.981 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.981 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.981 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.981 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.981 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339308+0000", 2024-04-26T01:08:08.981 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.981 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.981 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.981 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.981 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339308+0000", 2024-04-26T01:08:08.981 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.981 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.981 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.981 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.982 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339310+0000", 2024-04-26T01:08:08.982 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4959999999999999e-06 2024-04-26T01:08:08.982 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.982 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.982 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.982 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339328+0000", 2024-04-26T01:08:08.982 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8349999999999999e-05 2024-04-26T01:08:08.982 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.982 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.982 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.982 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339614+0000", 2024-04-26T01:08:08.982 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028623 2024-04-26T01:08:08.982 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.982 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.983 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.983 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339629+0000", 2024-04-26T01:08:08.983 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4803999999999999e-05 2024-04-26T01:08:08.983 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.983 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.983 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.983 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.340313+0000", 2024-04-26T01:08:08.983 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00068382200000000005 2024-04-26T01:08:08.983 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.983 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.983 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.983 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.340400+0000", 2024-04-26T01:08:08.983 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.7321e-05 2024-04-26T01:08:08.983 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.984 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.984 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.984 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346462+0000", 2024-04-26T01:08:08.984 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0060617939999999997 2024-04-26T01:08:08.984 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.984 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.984 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.984 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346517+0000", 2024-04-26T01:08:08.984 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4639000000000003e-05 2024-04-26T01:08:08.984 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.984 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.984 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.984 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.984 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.984 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:162 3.ds0 3:b1366e28:::smithi16428812-12:head [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:08.985 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.497135+0000", 2024-04-26T01:08:08.985 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.4030648079999999, 2024-04-26T01:08:08.985 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0049100000000000003, 2024-04-26T01:08:08.985 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.985 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.985 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:08.985 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:08.985 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:08.985 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:08.985 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 162 2024-04-26T01:08:08.985 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.985 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.985 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.985 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.985 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.497135+0000", 2024-04-26T01:08:08.986 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.986 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.986 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.986 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.986 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.497135+0000", 2024-04-26T01:08:08.986 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.986 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.986 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.986 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.986 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.497136+0000", 2024-04-26T01:08:08.986 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8800000000000001e-07 2024-04-26T01:08:08.986 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.986 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.987 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.987 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.497141+0000", 2024-04-26T01:08:08.987 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5559999999999998e-06 2024-04-26T01:08:08.987 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.987 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.987 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.987 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.497153+0000", 2024-04-26T01:08:08.987 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1934e-05 2024-04-26T01:08:08.987 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.987 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.987 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.987 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.497163+0000", 2024-04-26T01:08:08.987 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3479999999999993e-06 2024-04-26T01:08:08.987 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.987 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.988 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.988 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.500576+0000", 2024-04-26T01:08:08.988 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0034128880000000002 2024-04-26T01:08:08.988 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.988 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.988 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:08.988 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.500685+0000", 2024-04-26T01:08:08.988 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000109771 2024-04-26T01:08:08.988 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.988 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.988 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.988 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.502045+0000", 2024-04-26T01:08:08.988 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001359615 2024-04-26T01:08:08.988 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.988 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.989 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.989 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.989 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.989 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/30 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=38'19, trim_to=0'0, roll_forward_to=37'17))", 2024-04-26T01:08:08.989 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:04.427965+0000", 2024-04-26T01:08:08.989 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 4.472234898, 2024-04-26T01:08:08.989 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0080739560000000002, 2024-04-26T01:08:08.989 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:08.989 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:08.989 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:08.989 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:08.989 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.989 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:08.989 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.427965+0000", 2024-04-26T01:08:08.989 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.990 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.990 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.990 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:08.990 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.427965+0000", 2024-04-26T01:08:08.990 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:08.990 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.990 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.990 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:08.990 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.427966+0000", 2024-04-26T01:08:08.990 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3600000000000003e-07 2024-04-26T01:08:08.990 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.990 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.990 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:08.990 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.428022+0000", 2024-04-26T01:08:08.991 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6805999999999998e-05 2024-04-26T01:08:08.991 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.991 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.991 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:08.991 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.428186+0000", 2024-04-26T01:08:08.991 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000163287 2024-04-26T01:08:08.991 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.991 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.991 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:08.991 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.428212+0000", 2024-04-26T01:08:08.991 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6163e-05 2024-04-26T01:08:08.991 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.991 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.992 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:08.992 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.430074+0000", 2024-04-26T01:08:08.992 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0018619369999999999 2024-04-26T01:08:08.992 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.992 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.992 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:08.992 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.430098+0000", 2024-04-26T01:08:08.992 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.393e-05 2024-04-26T01:08:08.992 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.992 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.992 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:08.992 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.435960+0000", 2024-04-26T01:08:08.992 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0058619930000000002 2024-04-26T01:08:08.993 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:08.993 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:08.993 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:08.993 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.436039+0000", 2024-04-26T01:08:08.993 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9103999999999993e-05 2024-04-26T01:08:08.993 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.993 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.993 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.993 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:08.993 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:08.993 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:08.994 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:08:09.042 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:09.044+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:09.100 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:09.096+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:09.122 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:09.122 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:09.122 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:08:09.123 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:09.138 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:08:09.200 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:09.197+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:09.282 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-26T01:08:09.282 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T01:08:09.282 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-04-26T01:08:09.282 INFO:tasks.thrashosds.thrasher:Killing osd 11, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-26T01:08:09.283 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-04-26T01:08:09.283 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:08:09.301 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:09.301+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:09.326 INFO:teuthology.orchestra.run.smithi164.stderr:admin_socket: exception getting command descriptions: exception: [Errno 104] Connection reset by peer 2024-04-26T01:08:09.329 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-26T01:08:09.329 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:08:09.340 INFO:tasks.ceph.osd.11:Stopped 2024-04-26T01:08:09.340 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 11 2024-04-26T01:08:09.392 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-04-26T01:08:09.395 INFO:tasks.rados.rados.0.smithi164.stdout:141: done (1 left) 2024-04-26T01:08:09.395 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 34 current snap is 5 2024-04-26T01:08:09.395 INFO:tasks.rados.rados.0.smithi164.stdout:142: seq_num 78 ranges {557056=557056,1114112=671744,1785856=98304} 2024-04-26T01:08:09.402 INFO:tasks.rados.rados.0.smithi164.stdout:142: writing smithi16428812-34 from 557056 to 1114112 tid 1 2024-04-26T01:08:09.407 INFO:tasks.rados.rados.0.smithi164.stdout:142: writing smithi16428812-34 from 1114112 to 1785856 tid 2 2024-04-26T01:08:09.408 INFO:tasks.rados.rados.0.smithi164.stdout:142: writing smithi16428812-34 from 1785856 to 1884160 tid 3 2024-04-26T01:08:09.409 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 44 current snap is 5 2024-04-26T01:08:09.409 INFO:tasks.rados.rados.0.smithi164.stdout:143: seq_num 79 ranges {737280=573440,1310720=425984,1736704=344064} 2024-04-26T01:08:09.411 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:09.401+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:09.417 INFO:tasks.rados.rados.0.smithi164.stdout:143: writing smithi16428812-44 from 737280 to 1310720 tid 1 2024-04-26T01:08:09.420 INFO:tasks.rados.rados.0.smithi164.stdout:143: writing smithi16428812-44 from 1310720 to 1736704 tid 2 2024-04-26T01:08:09.423 INFO:tasks.rados.rados.0.smithi164.stdout:143: writing smithi16428812-44 from 1736704 to 2080768 tid 3 2024-04-26T01:08:09.423 INFO:tasks.rados.rados.0.smithi164.stdout:144: setattr oid 5 current snap is 5 2024-04-26T01:08:09.423 INFO:tasks.rados.rados.0.smithi164.stdout:140: finishing write tid 1 to smithi16428812-10 2024-04-26T01:08:09.423 INFO:tasks.rados.rados.0.smithi164.stdout:140: finishing write tid 2 to smithi16428812-10 2024-04-26T01:08:09.423 INFO:tasks.rados.rados.0.smithi164.stdout:140: finishing write tid 3 to smithi16428812-10 2024-04-26T01:08:09.424 INFO:tasks.rados.rados.0.smithi164.stdout:140: finishing write tid 4 to smithi16428812-10 2024-04-26T01:08:09.424 INFO:tasks.rados.rados.0.smithi164.stdout:140: finishing write tid 5 to smithi16428812-10 2024-04-26T01:08:09.424 INFO:tasks.rados.rados.0.smithi164.stdout:140: oid 10 updating version 0 to 28 2024-04-26T01:08:09.424 INFO:tasks.rados.rados.0.smithi164.stdout:140: oid 10 version 28 is already newer than 25 2024-04-26T01:08:09.424 INFO:tasks.rados.rados.0.smithi164.stdout:140: oid 10 version 28 is already newer than 27 2024-04-26T01:08:09.424 INFO:tasks.rados.rados.0.smithi164.stdout:140: oid 10 version 28 is already newer than 26 2024-04-26T01:08:09.424 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 10 v 28 (ObjNum 77 snap 5 seq_num 77) dirty exists 2024-04-26T01:08:09.424 INFO:tasks.rados.rados.0.smithi164.stdout:140: left oid 10 (ObjNum 77 snap 5 seq_num 77) 2024-04-26T01:08:09.424 INFO:tasks.rados.rados.0.smithi164.stdout:143: finishing write tid 1 to smithi16428812-44 2024-04-26T01:08:09.426 INFO:tasks.rados.rados.0.smithi164.stdout:140: done (3 left) 2024-04-26T01:08:09.426 INFO:tasks.rados.rados.0.smithi164.stdout:145: copy_from oid 39 from oid 30 current snap is 5 2024-04-26T01:08:09.426 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 35 current snap is 5 2024-04-26T01:08:09.426 INFO:tasks.rados.rados.0.smithi164.stdout:146: seq_num 80 ranges {0=589824,589824=573440,1163264=196608} 2024-04-26T01:08:09.430 INFO:tasks.rados.rados.0.smithi164.stdout:146: writing smithi16428812-35 from 0 to 589824 tid 1 2024-04-26T01:08:09.435 INFO:tasks.rados.rados.0.smithi164.stdout:146: writing smithi16428812-35 from 589824 to 1163264 tid 2 2024-04-26T01:08:09.437 INFO:tasks.rados.rados.0.smithi164.stdout:146: writing smithi16428812-35 from 1163264 to 1359872 tid 3 2024-04-26T01:08:09.437 INFO:tasks.rados.rados.0.smithi164.stdout:147: read oid 24 snap -1 2024-04-26T01:08:09.437 INFO:tasks.rados.rados.0.smithi164.stdout:147: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-26T01:08:09.437 INFO:tasks.rados.rados.0.smithi164.stdout:148: copy_from oid 47 from oid 12 current snap is 5 2024-04-26T01:08:09.437 INFO:tasks.rados.rados.0.smithi164.stdout:143: finishing write tid 2 to smithi16428812-44 2024-04-26T01:08:09.438 INFO:tasks.rados.rados.0.smithi164.stdout:143: finishing write tid 3 to smithi16428812-44 2024-04-26T01:08:09.438 INFO:tasks.rados.rados.0.smithi164.stdout:143: finishing write tid 4 to smithi16428812-44 2024-04-26T01:08:09.438 INFO:tasks.rados.rados.0.smithi164.stdout:149: read oid 19 snap -1 2024-04-26T01:08:09.438 INFO:tasks.rados.rados.0.smithi164.stdout:149: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-26T01:08:09.438 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 33 current snap is 5 2024-04-26T01:08:09.438 INFO:tasks.rados.rados.0.smithi164.stdout:143: finishing write tid 5 to smithi16428812-44 2024-04-26T01:08:09.438 INFO:tasks.rados.rados.0.smithi164.stdout:143: oid 44 updating version 0 to 16 2024-04-26T01:08:09.438 INFO:tasks.rados.rados.0.smithi164.stdout:143: oid 44 updating version 16 to 18 2024-04-26T01:08:09.438 INFO:tasks.rados.rados.0.smithi164.stdout:143: oid 44 version 18 is already newer than 15 2024-04-26T01:08:09.438 INFO:tasks.rados.rados.0.smithi164.stdout:143: oid 44 version 18 is already newer than 17 2024-04-26T01:08:09.438 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 44 v 18 (ObjNum 79 snap 5 seq_num 79) dirty exists 2024-04-26T01:08:09.438 INFO:tasks.rados.rados.0.smithi164.stdout:143: left oid 44 (ObjNum 79 snap 5 seq_num 79) 2024-04-26T01:08:09.438 INFO:tasks.rados.rados.0.smithi164.stdout:150: seq_num 81 ranges {540672=606208,1146880=720896,1867776=49152} 2024-04-26T01:08:09.444 INFO:tasks.rados.rados.0.smithi164.stdout:150: writing smithi16428812-33 from 540672 to 1146880 tid 1 2024-04-26T01:08:09.450 INFO:tasks.rados.rados.0.smithi164.stdout:150: writing smithi16428812-33 from 1146880 to 1867776 tid 2 2024-04-26T01:08:09.451 INFO:tasks.rados.rados.0.smithi164.stdout:150: writing smithi16428812-33 from 1867776 to 1916928 tid 3 2024-04-26T01:08:09.451 INFO:tasks.rados.rados.0.smithi164.stdout:143: done (8 left) 2024-04-26T01:08:09.451 INFO:tasks.rados.rados.0.smithi164.stdout:151: read oid 4 snap -1 2024-04-26T01:08:09.451 INFO:tasks.rados.rados.0.smithi164.stdout:151: expect (ObjNum 62 snap 2 seq_num 62) 2024-04-26T01:08:09.451 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 15 current snap is 5 2024-04-26T01:08:09.451 INFO:tasks.rados.rados.0.smithi164.stdout:152: seq_num 82 ranges {245760=622592,868352=475136,1343488=294912} 2024-04-26T01:08:09.456 INFO:tasks.rados.rados.0.smithi164.stdout:152: writing smithi16428812-15 from 245760 to 868352 tid 1 2024-04-26T01:08:09.460 INFO:tasks.rados.rados.0.smithi164.stdout:152: writing smithi16428812-15 from 868352 to 1343488 tid 2 2024-04-26T01:08:09.463 INFO:tasks.rados.rados.0.smithi164.stdout:152: writing smithi16428812-15 from 1343488 to 1638400 tid 3 2024-04-26T01:08:09.463 INFO:tasks.rados.rados.0.smithi164.stdout:153: copy_from oid 36 from oid 41 current snap is 5 2024-04-26T01:08:09.463 INFO:tasks.rados.rados.0.smithi164.stdout:146: finishing write tid 1 to smithi16428812-35 2024-04-26T01:08:09.463 INFO:tasks.rados.rados.0.smithi164.stdout:147: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-26T01:08:09.464 INFO:teuthology.orchestra.run.smithi164.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-26T01:08:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-26T01:08:09.468 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:08:09.492 INFO:tasks.rados.rados.0.smithi164.stdout:146: finishing write tid 2 to smithi16428812-35 2024-04-26T01:08:09.492 INFO:tasks.rados.rados.0.smithi164.stdout:146: finishing write tid 3 to smithi16428812-35 2024-04-26T01:08:09.492 INFO:tasks.rados.rados.0.smithi164.stdout:146: finishing write tid 4 to smithi16428812-35 2024-04-26T01:08:09.492 INFO:tasks.rados.rados.0.smithi164.stdout:146: finishing write tid 5 to smithi16428812-35 2024-04-26T01:08:09.492 INFO:tasks.rados.rados.0.smithi164.stdout:146: oid 35 updating version 0 to 26 2024-04-26T01:08:09.492 INFO:tasks.rados.rados.0.smithi164.stdout:146: oid 35 version 26 is already newer than 25 2024-04-26T01:08:09.492 INFO:tasks.rados.rados.0.smithi164.stdout:146: oid 35 updating version 26 to 27 2024-04-26T01:08:09.492 INFO:tasks.rados.rados.0.smithi164.stdout:146: oid 35 version 27 is already newer than 24 2024-04-26T01:08:09.492 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 35 v 27 (ObjNum 80 snap 5 seq_num 80) dirty exists 2024-04-26T01:08:09.492 INFO:tasks.rados.rados.0.smithi164.stdout:146: left oid 35 (ObjNum 80 snap 5 seq_num 80) 2024-04-26T01:08:09.493 INFO:tasks.rados.rados.0.smithi164.stdout:152: finishing write tid 1 to smithi16428812-15 2024-04-26T01:08:09.493 INFO:tasks.rados.rados.0.smithi164.stdout:152: finishing write tid 2 to smithi16428812-15 2024-04-26T01:08:09.493 INFO:tasks.rados.rados.0.smithi164.stdout:152: finishing write tid 3 to smithi16428812-15 2024-04-26T01:08:09.493 INFO:tasks.rados.rados.0.smithi164.stdout:152: finishing write tid 4 to smithi16428812-15 2024-04-26T01:08:09.493 INFO:tasks.rados.rados.0.smithi164.stdout:152: finishing write tid 5 to smithi16428812-15 2024-04-26T01:08:09.493 INFO:tasks.rados.rados.0.smithi164.stdout:152: oid 15 updating version 0 to 17 2024-04-26T01:08:09.493 INFO:tasks.rados.rados.0.smithi164.stdout:152: oid 15 version 17 is already newer than 16 2024-04-26T01:08:09.493 INFO:tasks.rados.rados.0.smithi164.stdout:152: oid 15 version 17 is already newer than 14 2024-04-26T01:08:09.493 INFO:tasks.rados.rados.0.smithi164.stdout:152: oid 15 version 17 is already newer than 15 2024-04-26T01:08:09.493 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 15 v 17 (ObjNum 82 snap 5 seq_num 82) dirty exists 2024-04-26T01:08:09.493 INFO:tasks.rados.rados.0.smithi164.stdout:152: left oid 15 (ObjNum 82 snap 5 seq_num 82) 2024-04-26T01:08:09.493 INFO:tasks.rados.rados.0.smithi164.stdout:146: done (10 left) 2024-04-26T01:08:09.493 INFO:tasks.rados.rados.0.smithi164.stdout:147: done (9 left) 2024-04-26T01:08:09.493 INFO:tasks.rados.rados.0.smithi164.stdout:152: done (8 left) 2024-04-26T01:08:09.493 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 31 current snap is 5 2024-04-26T01:08:09.494 INFO:tasks.rados.rados.0.smithi164.stdout:154: seq_num 83 ranges {507904=638976,1146880=622592,1769472=147456} 2024-04-26T01:08:09.498 INFO:tasks.rados.rados.0.smithi164.stdout:154: writing smithi16428812-31 from 507904 to 1146880 tid 1 2024-04-26T01:08:09.502 INFO:tasks.rados.rados.0.smithi164.stdout:154: writing smithi16428812-31 from 1146880 to 1769472 tid 2 2024-04-26T01:08:09.502 INFO:tasks.rados.rados.0.smithi164.stdout:154: writing smithi16428812-31 from 1769472 to 1916928 tid 3 2024-04-26T01:08:09.502 INFO:tasks.rados.rados.0.smithi164.stdout:155: rmattr oid 25 current snap is 5 2024-04-26T01:08:09.502 INFO:tasks.rados.rados.0.smithi164.stdout:155: done (9 left) 2024-04-26T01:08:09.503 INFO:tasks.rados.rados.0.smithi164.stdout:156: setattr oid 25 current snap is 5 2024-04-26T01:08:09.504 INFO:tasks.rados.rados.0.smithi164.stdout:157: rollback oid 13 current snap is 5 2024-04-26T01:08:09.504 INFO:tasks.rados.rados.0.smithi164.stdout:157: done (10 left) 2024-04-26T01:08:09.505 INFO:tasks.rados.rados.0.smithi164.stdout:158: rmattr oid 38 current snap is 5 2024-04-26T01:08:09.505 INFO:tasks.rados.rados.0.smithi164.stdout:159: snap_create 2024-04-26T01:08:09.505 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 38 v 0 (ObjNum 75 snap 4 seq_num 75) dirty exists 2024-04-26T01:08:09.514 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:09.512+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:09.590 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:09.591 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:09.591 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-04-26T01:08:09.591 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:09.603 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:09.600+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:09.605 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:08:09.681 INFO:tasks.ceph.mon.a.smithi008.stderr:2024-04-26T01:08:09.676+0000 7f99831a5640 -1 mon.a@0(leader).osd e43 definitely_dead 0 2024-04-26T01:08:09.682 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11 is already down. 2024-04-26T01:08:09.701 INFO:tasks.thrashosds.thrasher:Removing osd 11, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-04-26T01:08:09.702 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 11 2024-04-26T01:08:09.704 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:09.701+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:09.766 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:09.767 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:09.767 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-04-26T01:08:09.767 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-04-26T01:08:09.767 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:09.781 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:08:09.805 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:09.804+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:09.906 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:09.904+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:09.988 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:09.988 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:09.988 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:09.988 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:09.988 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.988 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.7s3 33/30 ECSubWrite(tid=9, reqid=client.4521.0:38, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:09.988 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.325556+0000", 2024-04-26T01:08:09.989 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 10.659238986, 2024-04-26T01:08:09.989 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.028941706000000001, 2024-04-26T01:08:09.989 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:09.989 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:09.989 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:09.989 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:09.989 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.989 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:09.989 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325556+0000", 2024-04-26T01:08:09.989 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:09.989 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.989 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.990 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:09.990 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325556+0000", 2024-04-26T01:08:09.990 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:09.990 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.990 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.990 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:09.990 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325557+0000", 2024-04-26T01:08:09.990 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.221e-06 2024-04-26T01:08:09.990 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.990 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.990 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:09.990 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325581+0000", 2024-04-26T01:08:09.991 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3502e-05 2024-04-26T01:08:09.991 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.991 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.991 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:09.991 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353123+0000", 2024-04-26T01:08:09.991 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.027542540000000001 2024-04-26T01:08:09.991 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.991 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.991 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:09.991 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353131+0000", 2024-04-26T01:08:09.991 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6909999999999997e-06 2024-04-26T01:08:09.992 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.992 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.992 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:09.992 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353823+0000", 2024-04-26T01:08:09.992 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00069229100000000004 2024-04-26T01:08:09.992 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.992 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.992 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:09.992 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353835+0000", 2024-04-26T01:08:09.992 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1507e-05 2024-04-26T01:08:09.992 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.992 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.992 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:09.993 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.354460+0000", 2024-04-26T01:08:09.993 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00062488599999999995 2024-04-26T01:08:09.993 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.993 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.993 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:09.993 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.354498+0000", 2024-04-26T01:08:09.993 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8068000000000002e-05 2024-04-26T01:08:09.993 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:09.993 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:09.993 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:09.993 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.994 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.994 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:43 3.6s0 3:61b5d9f4:::smithi16428812-15:head [append 0~245760,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:09.994 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.344935+0000", 2024-04-26T01:08:09.994 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 10.639860197000001, 2024-04-26T01:08:09.994 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012248536000000001, 2024-04-26T01:08:09.994 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:09.994 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:09.994 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:09.994 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:09.994 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:09.994 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:09.994 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 43 2024-04-26T01:08:09.995 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.995 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:09.995 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.995 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:09.995 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.344935+0000", 2024-04-26T01:08:09.995 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:09.995 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.995 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.995 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:09.995 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.344935+0000", 2024-04-26T01:08:09.995 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:09.996 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.996 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.996 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:09.996 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.344937+0000", 2024-04-26T01:08:09.996 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9750000000000001e-06 2024-04-26T01:08:09.996 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.996 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.996 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:09.996 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345580+0000", 2024-04-26T01:08:09.996 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00064293 2024-04-26T01:08:09.996 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.996 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.997 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:09.997 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345596+0000", 2024-04-26T01:08:09.997 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6334e-05 2024-04-26T01:08:09.997 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.997 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.997 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:09.997 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345605+0000", 2024-04-26T01:08:09.997 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2080000000000006e-06 2024-04-26T01:08:09.997 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.997 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.997 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:09.997 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345700+0000", 2024-04-26T01:08:09.997 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4698999999999996e-05 2024-04-26T01:08:09.998 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.998 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.998 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:09.998 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345954+0000", 2024-04-26T01:08:09.998 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00025357 2024-04-26T01:08:09.998 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.998 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.998 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:09.998 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.347095+0000", 2024-04-26T01:08:09.998 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001140978 2024-04-26T01:08:09.998 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.999 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.999 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:09.999 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349634+0000", 2024-04-26T01:08:09.999 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0025389969999999999 2024-04-26T01:08:09.999 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.999 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.999 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:09.999 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.357101+0000", 2024-04-26T01:08:09.999 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0074677579999999997 2024-04-26T01:08:09.999 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:09.999 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:09.999 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.000 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.357183+0000", 2024-04-26T01:08:10.000 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2087000000000002e-05 2024-04-26T01:08:10.000 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.000 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.000 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.000 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.000 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.000 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:44 3.6s0 3:61b5d9f4:::smithi16428812-15:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.000 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.345630+0000", 2024-04-26T01:08:10.000 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 10.639164846, 2024-04-26T01:08:10.000 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011979597999999999, 2024-04-26T01:08:10.000 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.001 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.001 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:10.001 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.001 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.001 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.001 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 44 2024-04-26T01:08:10.001 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.001 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.001 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.001 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.001 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345630+0000", 2024-04-26T01:08:10.001 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.002 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.002 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.002 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.002 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345630+0000", 2024-04-26T01:08:10.002 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.002 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.002 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.002 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.002 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345631+0000", 2024-04-26T01:08:10.002 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7999999999999995e-07 2024-04-26T01:08:10.002 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.002 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.003 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.003 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345634+0000", 2024-04-26T01:08:10.003 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.32e-06 2024-04-26T01:08:10.003 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.003 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.003 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.003 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345643+0000", 2024-04-26T01:08:10.003 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2130000000000008e-06 2024-04-26T01:08:10.003 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.003 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.003 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.003 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345649+0000", 2024-04-26T01:08:10.004 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.127e-06 2024-04-26T01:08:10.004 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.004 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.004 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.004 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.348399+0000", 2024-04-26T01:08:10.004 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027505519999999999 2024-04-26T01:08:10.004 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.004 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.004 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.004 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.348498+0000", 2024-04-26T01:08:10.004 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9272999999999996e-05 2024-04-26T01:08:10.005 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.005 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.005 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.005 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.348992+0000", 2024-04-26T01:08:10.005 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000493444 2024-04-26T01:08:10.005 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.005 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.005 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.005 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349687+0000", 2024-04-26T01:08:10.006 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00069528500000000002 2024-04-26T01:08:10.006 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.006 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.006 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:10.006 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.357516+0000", 2024-04-26T01:08:10.006 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0078283520000000002 2024-04-26T01:08:10.006 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.006 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.006 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.006 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.357610+0000", 2024-04-26T01:08:10.006 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4351999999999994e-05 2024-04-26T01:08:10.007 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.007 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.007 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.007 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.007 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.007 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:45 3.6s0 3:61b5d9f4:::smithi16428812-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.007 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.345679+0000", 2024-04-26T01:08:10.007 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 10.639116136, 2024-04-26T01:08:10.007 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014982359000000001, 2024-04-26T01:08:10.007 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.007 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.008 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.008 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.008 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.008 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.008 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 45 2024-04-26T01:08:10.008 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.008 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.008 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.008 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.008 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345679+0000", 2024-04-26T01:08:10.008 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.008 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.009 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.009 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.009 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345679+0000", 2024-04-26T01:08:10.009 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.009 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.009 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.009 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.009 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345679+0000", 2024-04-26T01:08:10.009 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9599999999999999e-07 2024-04-26T01:08:10.009 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.009 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.010 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.010 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345683+0000", 2024-04-26T01:08:10.010 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0110000000000001e-06 2024-04-26T01:08:10.010 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.010 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.010 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.010 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345696+0000", 2024-04-26T01:08:10.010 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3557000000000001e-05 2024-04-26T01:08:10.010 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.010 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.010 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.011 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345701+0000", 2024-04-26T01:08:10.011 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6190000000000002e-06 2024-04-26T01:08:10.011 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.011 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.011 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.011 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349537+0000", 2024-04-26T01:08:10.011 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0038363500000000001 2024-04-26T01:08:10.011 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.011 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.011 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:10.011 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349626+0000", 2024-04-26T01:08:10.011 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8757000000000004e-05 2024-04-26T01:08:10.012 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.012 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.012 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.012 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.358038+0000", 2024-04-26T01:08:10.012 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0084123970000000003 2024-04-26T01:08:10.012 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.012 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.012 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.012 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.358133+0000", 2024-04-26T01:08:10.012 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4949000000000002e-05 2024-04-26T01:08:10.012 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.013 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.013 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.013 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.360661+0000", 2024-04-26T01:08:10.013 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002528223 2024-04-26T01:08:10.013 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.013 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.014 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.014 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.014 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.014 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.fs6 33/30 ECSubWrite(tid=13, reqid=client.4521.0:53, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:10.014 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.352280+0000", 2024-04-26T01:08:10.014 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 10.632515547000001, 2024-04-26T01:08:10.014 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.017609129000000001, 2024-04-26T01:08:10.014 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.014 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.014 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:10.014 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.015 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.015 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.015 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.352280+0000", 2024-04-26T01:08:10.015 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.015 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.015 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.015 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.015 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.352280+0000", 2024-04-26T01:08:10.015 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.015 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.015 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.016 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.016 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.352280+0000", 2024-04-26T01:08:10.016 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6400000000000001e-07 2024-04-26T01:08:10.016 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.016 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.016 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.016 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.352302+0000", 2024-04-26T01:08:10.016 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1577e-05 2024-04-26T01:08:10.016 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.016 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.016 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.016 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369217+0000", 2024-04-26T01:08:10.016 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.016914825000000001 2024-04-26T01:08:10.017 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.017 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.017 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.017 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369224+0000", 2024-04-26T01:08:10.017 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9720000000000003e-06 2024-04-26T01:08:10.017 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.017 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.017 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.017 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369277+0000", 2024-04-26T01:08:10.017 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2924999999999999e-05 2024-04-26T01:08:10.017 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.017 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.018 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.018 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369284+0000", 2024-04-26T01:08:10.018 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0840000000000003e-06 2024-04-26T01:08:10.018 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.018 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.018 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.018 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369847+0000", 2024-04-26T01:08:10.018 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00056294600000000004 2024-04-26T01:08:10.018 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.018 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.018 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.018 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369889+0000", 2024-04-26T01:08:10.019 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1936e-05 2024-04-26T01:08:10.019 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.019 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.019 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.019 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.019 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.019 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.7s3 33/30 ECSubWrite(tid=16, reqid=client.4521.0:65, at_version=33'8, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:08:10.019 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.366487+0000", 2024-04-26T01:08:10.019 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 10.618307659999999, 2024-04-26T01:08:10.019 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.079262088999999994, 2024-04-26T01:08:10.019 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.019 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.019 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:10.020 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.020 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.020 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.020 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366487+0000", 2024-04-26T01:08:10.020 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.020 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.020 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.020 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.020 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366487+0000", 2024-04-26T01:08:10.020 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.020 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.020 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.020 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.021 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366488+0000", 2024-04-26T01:08:10.021 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6700000000000002e-07 2024-04-26T01:08:10.021 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.021 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.021 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.021 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366510+0000", 2024-04-26T01:08:10.021 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1591999999999999e-05 2024-04-26T01:08:10.021 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.021 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.021 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.021 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444763+0000", 2024-04-26T01:08:10.021 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.078252639999999998 2024-04-26T01:08:10.022 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.022 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.022 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.022 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444773+0000", 2024-04-26T01:08:10.022 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0134e-05 2024-04-26T01:08:10.022 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.022 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.022 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.022 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444870+0000", 2024-04-26T01:08:10.022 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6847000000000003e-05 2024-04-26T01:08:10.022 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.023 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.023 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.023 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444880+0000", 2024-04-26T01:08:10.023 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0312999999999999e-05 2024-04-26T01:08:10.023 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.023 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.023 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.023 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.445715+0000", 2024-04-26T01:08:10.023 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00083498700000000003 2024-04-26T01:08:10.023 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.023 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.023 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.023 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.445750+0000", 2024-04-26T01:08:10.024 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4609e-05 2024-04-26T01:08:10.024 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.024 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.024 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.024 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.024 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.024 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:70 3.6s0 3:6387e095:::smithi16428812-24:head [append 0~393216,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.024 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.371929+0000", 2024-04-26T01:08:10.024 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 10.612866359, 2024-04-26T01:08:10.024 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0085695949999999993, 2024-04-26T01:08:10.024 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.024 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.025 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:10.025 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.025 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.025 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.025 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 70 2024-04-26T01:08:10.025 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.025 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.025 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.025 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.025 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.371929+0000", 2024-04-26T01:08:10.025 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.025 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.026 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.026 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.026 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.371929+0000", 2024-04-26T01:08:10.026 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.026 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.026 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.026 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.026 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.371931+0000", 2024-04-26T01:08:10.026 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.232e-06 2024-04-26T01:08:10.026 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.026 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.026 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.027 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372322+0000", 2024-04-26T01:08:10.027 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00039080799999999998 2024-04-26T01:08:10.027 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.027 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.027 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.027 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372337+0000", 2024-04-26T01:08:10.027 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5362999999999998e-05 2024-04-26T01:08:10.027 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.027 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.027 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.027 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372347+0000", 2024-04-26T01:08:10.027 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9040000000000004e-06 2024-04-26T01:08:10.028 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.028 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.028 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.028 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372407+0000", 2024-04-26T01:08:10.028 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9570000000000001e-05 2024-04-26T01:08:10.028 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.028 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.028 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.028 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372519+0000", 2024-04-26T01:08:10.028 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000112693 2024-04-26T01:08:10.028 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.028 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.028 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.029 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.373257+0000", 2024-04-26T01:08:10.029 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00073773999999999997 2024-04-26T01:08:10.029 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.029 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.029 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.029 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.375196+0000", 2024-04-26T01:08:10.029 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001939203 2024-04-26T01:08:10.029 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.029 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.029 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:10.029 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380381+0000", 2024-04-26T01:08:10.029 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0051844689999999997 2024-04-26T01:08:10.029 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.029 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.030 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.030 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380498+0000", 2024-04-26T01:08:10.030 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000117613 2024-04-26T01:08:10.030 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.030 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.030 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.030 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.030 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.030 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:71 3.6s0 3:6387e095:::smithi16428812-24:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.030 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.372368+0000", 2024-04-26T01:08:10.030 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 10.612427576, 2024-04-26T01:08:10.030 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0085830960000000001, 2024-04-26T01:08:10.031 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.031 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.031 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:10.031 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.031 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.031 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.031 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 71 2024-04-26T01:08:10.031 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.031 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.031 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.031 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.032 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372368+0000", 2024-04-26T01:08:10.032 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.032 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.032 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.032 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.032 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372368+0000", 2024-04-26T01:08:10.032 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.032 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.032 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.032 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.032 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372368+0000", 2024-04-26T01:08:10.033 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0999999999999997e-07 2024-04-26T01:08:10.033 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.033 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.033 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.033 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372373+0000", 2024-04-26T01:08:10.033 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7650000000000001e-06 2024-04-26T01:08:10.033 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.033 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.033 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.033 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372382+0000", 2024-04-26T01:08:10.033 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6629999999999999e-06 2024-04-26T01:08:10.034 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.034 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.034 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.034 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372387+0000", 2024-04-26T01:08:10.034 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3720000000000004e-06 2024-04-26T01:08:10.034 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.034 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.034 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.034 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.373890+0000", 2024-04-26T01:08:10.034 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015026569999999999 2024-04-26T01:08:10.034 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.035 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.035 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.035 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374039+0000", 2024-04-26T01:08:10.035 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000149486 2024-04-26T01:08:10.035 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.035 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.035 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.035 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374554+0000", 2024-04-26T01:08:10.035 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00051440600000000002 2024-04-26T01:08:10.035 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.036 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.036 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.036 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.375476+0000", 2024-04-26T01:08:10.036 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000922484 2024-04-26T01:08:10.036 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.036 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.036 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:10.036 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380847+0000", 2024-04-26T01:08:10.036 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0053703809999999996 2024-04-26T01:08:10.036 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.036 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.036 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.037 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380951+0000", 2024-04-26T01:08:10.037 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000103972 2024-04-26T01:08:10.037 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.037 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.037 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.037 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.037 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.037 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:72 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.037 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.372400+0000", 2024-04-26T01:08:10.037 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 10.612395131, 2024-04-26T01:08:10.037 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018232822999999999, 2024-04-26T01:08:10.037 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.038 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.038 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.038 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.038 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.038 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.038 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 72 2024-04-26T01:08:10.038 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.038 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.038 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.038 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.038 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372400+0000", 2024-04-26T01:08:10.038 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.039 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.039 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.039 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.039 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372400+0000", 2024-04-26T01:08:10.039 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.039 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.039 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.039 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.039 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372401+0000", 2024-04-26T01:08:10.039 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7899999999999998e-07 2024-04-26T01:08:10.039 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.039 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.039 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.040 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372404+0000", 2024-04-26T01:08:10.040 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0469999999999998e-06 2024-04-26T01:08:10.040 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.040 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.040 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.040 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372413+0000", 2024-04-26T01:08:10.040 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3729999999999999e-06 2024-04-26T01:08:10.040 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.040 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.040 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.040 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372419+0000", 2024-04-26T01:08:10.040 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1070000000000001e-06 2024-04-26T01:08:10.041 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.041 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.041 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.041 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374838+0000", 2024-04-26T01:08:10.041 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024188289999999999 2024-04-26T01:08:10.041 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.041 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.041 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:10.041 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374919+0000", 2024-04-26T01:08:10.041 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0717000000000007e-05 2024-04-26T01:08:10.041 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.041 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.041 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.042 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.381324+0000", 2024-04-26T01:08:10.042 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.006405666 2024-04-26T01:08:10.042 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.042 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.042 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.042 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.381402+0000", 2024-04-26T01:08:10.042 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7323000000000005e-05 2024-04-26T01:08:10.042 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.042 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.042 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.042 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.390633+0000", 2024-04-26T01:08:10.043 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0092311819999999992 2024-04-26T01:08:10.043 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.043 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.043 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.043 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.043 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.043 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpRead(3.cs2 33/30 ECSubRead(tid=19, to_read={3:3ca9ce7a:::smithi16428812-23:head=0,4096,0}, subchunks={3:3ca9ce7a:::smithi16428812-23:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:10.043 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.374332+0000", 2024-04-26T01:08:10.043 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 10.610463038000001, 2024-04-26T01:08:10.043 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.023990841999999998, 2024-04-26T01:08:10.043 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.043 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.043 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:10.044 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.044 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.044 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.044 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374332+0000", 2024-04-26T01:08:10.044 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.044 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.044 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.044 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.044 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374332+0000", 2024-04-26T01:08:10.044 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.044 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.044 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.044 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.045 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374333+0000", 2024-04-26T01:08:10.045 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0070000000000001e-06 2024-04-26T01:08:10.045 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.045 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.045 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.045 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374345+0000", 2024-04-26T01:08:10.045 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1853e-05 2024-04-26T01:08:10.045 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.045 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.045 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.045 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398055+0000", 2024-04-26T01:08:10.045 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.023710078999999998 2024-04-26T01:08:10.045 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.045 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.046 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.046 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398067+0000", 2024-04-26T01:08:10.046 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2455e-05 2024-04-26T01:08:10.046 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.046 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.046 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.046 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398160+0000", 2024-04-26T01:08:10.046 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2682000000000005e-05 2024-04-26T01:08:10.046 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.046 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.046 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.046 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398323+0000", 2024-04-26T01:08:10.047 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016276600000000001 2024-04-26T01:08:10.047 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.047 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.047 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.047 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.047 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.047 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs2 33/30 ECSubWrite(tid=21, reqid=client.4521.0:89, at_version=33'8, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:08:10.047 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.399506+0000", 2024-04-26T01:08:10.047 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 10.585288648000001, 2024-04-26T01:08:10.047 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012907333999999999, 2024-04-26T01:08:10.047 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.047 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.047 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:10.048 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.048 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.048 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.048 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.399506+0000", 2024-04-26T01:08:10.048 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.048 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.048 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.048 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.048 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.399506+0000", 2024-04-26T01:08:10.048 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.048 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.048 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.048 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.048 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.399507+0000", 2024-04-26T01:08:10.049 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0800000000000004e-07 2024-04-26T01:08:10.049 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.049 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.049 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.049 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.399525+0000", 2024-04-26T01:08:10.049 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7342000000000001e-05 2024-04-26T01:08:10.049 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.049 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.049 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.049 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411386+0000", 2024-04-26T01:08:10.049 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011861772 2024-04-26T01:08:10.049 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.049 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.050 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.050 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411395+0000", 2024-04-26T01:08:10.050 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0019999999999995e-06 2024-04-26T01:08:10.050 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.050 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.050 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.050 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411484+0000", 2024-04-26T01:08:10.050 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8993000000000007e-05 2024-04-26T01:08:10.050 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.050 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.050 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.050 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411495+0000", 2024-04-26T01:08:10.051 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1182000000000001e-05 2024-04-26T01:08:10.051 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.051 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.051 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.051 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.412364+0000", 2024-04-26T01:08:10.051 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00086887599999999998 2024-04-26T01:08:10.051 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.051 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.051 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.051 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.412414+0000", 2024-04-26T01:08:10.051 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9459000000000002e-05 2024-04-26T01:08:10.051 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.051 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.052 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.052 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.052 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.052 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.fs6 33/30 ECSubWrite(tid=28, reqid=client.4521.0:107, at_version=33'6, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:08:10.052 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.426040+0000", 2024-04-26T01:08:10.052 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 10.558755162000001, 2024-04-26T01:08:10.052 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0087163850000000001, 2024-04-26T01:08:10.052 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.052 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.052 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:10.052 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.052 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.052 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.053 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.426040+0000", 2024-04-26T01:08:10.053 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.053 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.053 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.053 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.053 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.426040+0000", 2024-04-26T01:08:10.053 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.053 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.053 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.053 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.053 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.426041+0000", 2024-04-26T01:08:10.053 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5700000000000002e-07 2024-04-26T01:08:10.054 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.054 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.054 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.054 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.426061+0000", 2024-04-26T01:08:10.054 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.018e-05 2024-04-26T01:08:10.054 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.054 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.054 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.054 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.434007+0000", 2024-04-26T01:08:10.054 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0079456760000000005 2024-04-26T01:08:10.054 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.054 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.055 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.055 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.434026+0000", 2024-04-26T01:08:10.055 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9159000000000001e-05 2024-04-26T01:08:10.055 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.055 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.055 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.055 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.434098+0000", 2024-04-26T01:08:10.055 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2441999999999995e-05 2024-04-26T01:08:10.055 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.055 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.055 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.055 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.434108+0000", 2024-04-26T01:08:10.056 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9310000000000003e-06 2024-04-26T01:08:10.056 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.056 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.056 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.056 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.434726+0000", 2024-04-26T01:08:10.056 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00061808999999999996 2024-04-26T01:08:10.056 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.056 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.056 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.056 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.434756+0000", 2024-04-26T01:08:10.056 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0150000000000001e-05 2024-04-26T01:08:10.056 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.056 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.056 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.057 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.057 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.057 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:239 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~393216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:10.057 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.342338+0000", 2024-04-26T01:08:10.057 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 6.6424573589999998, 2024-04-26T01:08:10.057 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.019617285000000002, 2024-04-26T01:08:10.057 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.057 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.057 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.057 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.057 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.057 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.057 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 239 2024-04-26T01:08:10.057 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.058 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.058 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.058 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.058 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342338+0000", 2024-04-26T01:08:10.058 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.058 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.058 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.058 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.058 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342338+0000", 2024-04-26T01:08:10.058 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.058 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.058 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.059 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.059 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342340+0000", 2024-04-26T01:08:10.059 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4700000000000001e-06 2024-04-26T01:08:10.059 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.059 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.059 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.059 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342351+0000", 2024-04-26T01:08:10.059 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0925e-05 2024-04-26T01:08:10.059 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.059 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.059 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.059 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342379+0000", 2024-04-26T01:08:10.059 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8140000000000002e-05 2024-04-26T01:08:10.059 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.060 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.060 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.060 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342395+0000", 2024-04-26T01:08:10.060 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5620999999999998e-05 2024-04-26T01:08:10.060 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.060 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.060 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.060 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342550+0000", 2024-04-26T01:08:10.060 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015483000000000001 2024-04-26T01:08:10.060 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.060 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.060 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op must wait for map", 2024-04-26T01:08:10.060 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342566+0000", 2024-04-26T01:08:10.061 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6214000000000001e-05 2024-04-26T01:08:10.061 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.061 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.061 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.061 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.349020+0000", 2024-04-26T01:08:10.061 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0064544240000000003 2024-04-26T01:08:10.061 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.061 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.061 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.061 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.349117+0000", 2024-04-26T01:08:10.061 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6212000000000007e-05 2024-04-26T01:08:10.061 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.062 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.062 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.062 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.361955+0000", 2024-04-26T01:08:10.062 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012838449 2024-04-26T01:08:10.062 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.062 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.062 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.062 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.062 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.062 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:240 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:10.062 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.342427+0000", 2024-04-26T01:08:10.062 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 6.6423681490000002, 2024-04-26T01:08:10.063 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.019852153000000001, 2024-04-26T01:08:10.063 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.063 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.063 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.063 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.063 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.063 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.063 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 240 2024-04-26T01:08:10.063 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.063 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.063 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.063 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.063 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342427+0000", 2024-04-26T01:08:10.064 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.064 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.064 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.064 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.064 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342427+0000", 2024-04-26T01:08:10.064 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.064 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.064 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.064 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.064 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342428+0000", 2024-04-26T01:08:10.064 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2200000000000003e-07 2024-04-26T01:08:10.064 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.064 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.064 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.065 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342435+0000", 2024-04-26T01:08:10.065 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0180000000000004e-06 2024-04-26T01:08:10.065 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.065 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.065 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.065 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342451+0000", 2024-04-26T01:08:10.065 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6011e-05 2024-04-26T01:08:10.065 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.065 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.065 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.065 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342459+0000", 2024-04-26T01:08:10.065 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1370000000000002e-06 2024-04-26T01:08:10.065 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.066 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.066 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.066 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342766+0000", 2024-04-26T01:08:10.066 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00030684000000000001 2024-04-26T01:08:10.066 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.066 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.066 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:10.066 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342784+0000", 2024-04-26T01:08:10.066 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7773000000000001e-05 2024-04-26T01:08:10.066 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.066 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.066 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.067 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.349542+0000", 2024-04-26T01:08:10.067 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0067586349999999998 2024-04-26T01:08:10.067 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.067 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.067 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.067 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.349627+0000", 2024-04-26T01:08:10.067 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4790999999999995e-05 2024-04-26T01:08:10.067 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.067 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.067 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.067 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.362279+0000", 2024-04-26T01:08:10.067 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012652126 2024-04-26T01:08:10.067 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.068 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.068 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.068 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.068 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.068 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:241 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:10.068 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.342661+0000", 2024-04-26T01:08:10.068 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 6.6421342990000003, 2024-04-26T01:08:10.068 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.020164171000000002, 2024-04-26T01:08:10.068 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.068 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.068 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.068 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.068 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.069 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.069 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 241 2024-04-26T01:08:10.069 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.069 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.069 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.069 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.069 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342661+0000", 2024-04-26T01:08:10.069 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.069 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.069 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.069 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.069 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342661+0000", 2024-04-26T01:08:10.069 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.069 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.070 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.070 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.070 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342662+0000", 2024-04-26T01:08:10.070 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0100000000000003e-07 2024-04-26T01:08:10.070 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.070 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.070 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.070 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342668+0000", 2024-04-26T01:08:10.070 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2369999999999997e-06 2024-04-26T01:08:10.070 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.070 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.070 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.071 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342682+0000", 2024-04-26T01:08:10.071 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4166e-05 2024-04-26T01:08:10.071 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.071 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.071 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.071 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342691+0000", 2024-04-26T01:08:10.071 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8000000000000004e-06 2024-04-26T01:08:10.071 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.071 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.071 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.071 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342844+0000", 2024-04-26T01:08:10.071 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015316599999999999 2024-04-26T01:08:10.072 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.072 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.072 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:10.072 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342853+0000", 2024-04-26T01:08:10.072 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9700000000000005e-06 2024-04-26T01:08:10.072 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.072 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.072 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.072 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.350630+0000", 2024-04-26T01:08:10.072 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0077770890000000001 2024-04-26T01:08:10.072 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.073 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.073 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.073 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.350718+0000", 2024-04-26T01:08:10.073 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8156000000000001e-05 2024-04-26T01:08:10.073 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.073 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.073 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.073 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.362825+0000", 2024-04-26T01:08:10.073 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012106686 2024-04-26T01:08:10.073 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.073 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.074 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.074 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.074 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.074 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:276 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:10.074 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.413967+0000", 2024-04-26T01:08:10.074 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 5.5708276840000002, 2024-04-26T01:08:10.074 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014303402, 2024-04-26T01:08:10.074 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.074 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.074 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.075 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.075 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.075 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.075 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 276 2024-04-26T01:08:10.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.075 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.075 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.075 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413967+0000", 2024-04-26T01:08:10.075 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.075 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.075 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413967+0000", 2024-04-26T01:08:10.076 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.076 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.076 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.076 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.076 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413969+0000", 2024-04-26T01:08:10.076 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.24e-06 2024-04-26T01:08:10.076 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.076 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.076 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.076 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413996+0000", 2024-04-26T01:08:10.076 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7093999999999998e-05 2024-04-26T01:08:10.076 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414016+0000", 2024-04-26T01:08:10.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0072999999999999e-05 2024-04-26T01:08:10.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414031+0000", 2024-04-26T01:08:10.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5159e-05 2024-04-26T01:08:10.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.420581+0000", 2024-04-26T01:08:10.078 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0065497519999999998 2024-04-26T01:08:10.078 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.078 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.078 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.078 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.420691+0000", 2024-04-26T01:08:10.078 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000110259 2024-04-26T01:08:10.078 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.078 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.078 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.078 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.428271+0000", 2024-04-26T01:08:10.078 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0075798250000000001 2024-04-26T01:08:10.078 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.079 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.079 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.079 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.079 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.079 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs2 41/30 ECSubWrite(tid=91, reqid=client.4521.0:345, at_version=41'27, trim_to=0'0, roll_forward_to=41'26))", 2024-04-26T01:08:10.079 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:07.610935+0000", 2024-04-26T01:08:10.079 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.3738597759999998, 2024-04-26T01:08:10.079 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.629675231, 2024-04-26T01:08:10.079 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.079 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.079 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:10.079 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.079 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.080 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.080 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.610935+0000", 2024-04-26T01:08:10.080 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.080 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.080 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.080 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.080 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.610935+0000", 2024-04-26T01:08:10.080 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.080 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.080 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.080 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.081 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.610938+0000", 2024-04-26T01:08:10.081 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.39e-06 2024-04-26T01:08:10.081 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.081 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.081 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.081 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.611405+0000", 2024-04-26T01:08:10.081 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00046744899999999999 2024-04-26T01:08:10.081 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.081 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.081 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.082 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238064+0000", 2024-04-26T01:08:10.082 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.62665866100000001 2024-04-26T01:08:10.082 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.082 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.082 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.082 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238080+0000", 2024-04-26T01:08:10.082 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6623e-05 2024-04-26T01:08:10.082 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.082 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.082 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.082 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238214+0000", 2024-04-26T01:08:10.083 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013316900000000001 2024-04-26T01:08:10.083 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.083 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.083 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.083 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238235+0000", 2024-04-26T01:08:10.083 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0937e-05 2024-04-26T01:08:10.083 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.083 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.083 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.083 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.240510+0000", 2024-04-26T01:08:10.083 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022752860000000001 2024-04-26T01:08:10.084 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.084 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.084 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.084 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.240611+0000", 2024-04-26T01:08:10.084 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000100716 2024-04-26T01:08:10.084 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.084 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.084 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.084 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.084 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.084 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:390 3.6s0 3:61b5d9f4:::smithi16428812-15:head [append 0~294912] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.084 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.463190+0000", 2024-04-26T01:08:10.084 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.52160508100000003, 2024-04-26T01:08:10.084 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0086136270000000004, 2024-04-26T01:08:10.085 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.085 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.085 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:10.085 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.085 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.085 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.085 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 390 2024-04-26T01:08:10.085 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.085 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.085 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.085 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.085 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463190+0000", 2024-04-26T01:08:10.085 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.086 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.086 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.086 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.086 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463190+0000", 2024-04-26T01:08:10.086 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.086 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.086 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.086 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.086 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463192+0000", 2024-04-26T01:08:10.086 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.782e-06 2024-04-26T01:08:10.086 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.086 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.087 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.087 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463587+0000", 2024-04-26T01:08:10.087 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00039563800000000002 2024-04-26T01:08:10.087 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.087 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.087 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.087 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463603+0000", 2024-04-26T01:08:10.087 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5758999999999999e-05 2024-04-26T01:08:10.087 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.087 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.087 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.087 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463613+0000", 2024-04-26T01:08:10.087 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8980000000000008e-06 2024-04-26T01:08:10.088 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.088 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.088 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.088 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463684+0000", 2024-04-26T01:08:10.088 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1229999999999994e-05 2024-04-26T01:08:10.088 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.088 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.088 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.088 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463793+0000", 2024-04-26T01:08:10.088 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000108393 2024-04-26T01:08:10.088 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.088 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.089 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.089 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.464412+0000", 2024-04-26T01:08:10.089 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00061915099999999997 2024-04-26T01:08:10.089 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.089 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.089 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.089 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.466416+0000", 2024-04-26T01:08:10.089 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0020037369999999998 2024-04-26T01:08:10.089 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.089 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.089 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:10.089 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.471741+0000", 2024-04-26T01:08:10.089 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0053253450000000004 2024-04-26T01:08:10.089 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.090 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.090 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.090 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.471804+0000", 2024-04-26T01:08:10.090 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2694000000000007e-05 2024-04-26T01:08:10.090 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.090 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.090 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.090 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.090 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.090 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:391 3.6s0 3:61b5d9f4:::smithi16428812-15:head [setxattr _header (56) in=7b] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.090 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.463636+0000", 2024-04-26T01:08:10.090 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.52115944400000003, 2024-04-26T01:08:10.091 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0084033649999999994, 2024-04-26T01:08:10.091 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.091 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.091 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:10.091 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.091 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.091 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.091 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 391 2024-04-26T01:08:10.091 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.091 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.092 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.092 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.092 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463636+0000", 2024-04-26T01:08:10.092 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.092 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.092 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.092 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.092 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463636+0000", 2024-04-26T01:08:10.092 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.092 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.092 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.093 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.093 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463637+0000", 2024-04-26T01:08:10.093 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3e-06 2024-04-26T01:08:10.093 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.093 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.093 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.093 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463642+0000", 2024-04-26T01:08:10.093 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.0690000000000001e-06 2024-04-26T01:08:10.093 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.093 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.093 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.093 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463652+0000", 2024-04-26T01:08:10.093 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0419e-05 2024-04-26T01:08:10.093 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.094 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.094 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.094 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463661+0000", 2024-04-26T01:08:10.094 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2849999999999995e-06 2024-04-26T01:08:10.094 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.094 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.094 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.094 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.465196+0000", 2024-04-26T01:08:10.094 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015353949999999999 2024-04-26T01:08:10.094 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.094 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.095 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.095 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.465294+0000", 2024-04-26T01:08:10.095 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7548999999999997e-05 2024-04-26T01:08:10.095 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.095 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.095 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.095 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.465840+0000", 2024-04-26T01:08:10.095 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00054668200000000005 2024-04-26T01:08:10.095 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.095 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.095 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.095 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.467488+0000", 2024-04-26T01:08:10.096 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001647667 2024-04-26T01:08:10.096 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.096 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.096 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:10.096 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.471939+0000", 2024-04-26T01:08:10.096 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0044511259999999997 2024-04-26T01:08:10.096 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.096 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.096 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.096 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.472039+0000", 2024-04-26T01:08:10.096 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9872999999999997e-05 2024-04-26T01:08:10.096 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.097 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.097 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.097 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.097 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.097 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:392 3.6s0 3:61b5d9f4:::smithi16428812-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.097 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.463673+0000", 2024-04-26T01:08:10.097 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.52112164599999999, 2024-04-26T01:08:10.097 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010240143, 2024-04-26T01:08:10.097 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.097 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.097 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.097 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.098 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.098 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.098 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 392 2024-04-26T01:08:10.098 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.098 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.098 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.098 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.098 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463673+0000", 2024-04-26T01:08:10.098 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.098 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.098 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.098 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.098 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463673+0000", 2024-04-26T01:08:10.099 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.099 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.099 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.099 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.099 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463674+0000", 2024-04-26T01:08:10.099 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.9899999999999999e-07 2024-04-26T01:08:10.099 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.099 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.099 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.099 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463677+0000", 2024-04-26T01:08:10.099 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0460000000000001e-06 2024-04-26T01:08:10.099 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.099 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.100 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.100 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463686+0000", 2024-04-26T01:08:10.100 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1290000000000002e-06 2024-04-26T01:08:10.100 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.100 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.100 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.100 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463694+0000", 2024-04-26T01:08:10.100 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.4919999999999996e-06 2024-04-26T01:08:10.100 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.100 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.100 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.100 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.466138+0000", 2024-04-26T01:08:10.100 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024443120000000001 2024-04-26T01:08:10.100 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.101 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.101 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:10.101 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.466255+0000", 2024-04-26T01:08:10.101 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000116246 2024-04-26T01:08:10.101 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.101 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.101 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.101 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.472474+0000", 2024-04-26T01:08:10.101 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0062197420000000003 2024-04-26T01:08:10.101 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.101 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.101 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.101 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.472536+0000", 2024-04-26T01:08:10.102 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2008000000000007e-05 2024-04-26T01:08:10.102 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.102 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.102 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.102 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.473914+0000", 2024-04-26T01:08:10.102 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001377369 2024-04-26T01:08:10.102 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.102 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.102 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.102 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.102 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.102 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:10.104 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:08:10.104 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:10.100+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:10.106 INFO:tasks.ceph.mon.a.smithi008.stderr:2024-04-26T01:08:10.104+0000 7f99831a5640 -1 mon.a@0(leader).osd e43 definitely_dead 0 2024-04-26T01:08:10.205 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:10.201+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:10.224 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:10.225 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:10.225 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.225 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:378 3.7s0 3.86a9bb67 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.225 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.436792+0000", 2024-04-26T01:08:10.225 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.78657793399999998, 2024-04-26T01:08:10.225 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.78660682400000004, 2024-04-26T01:08:10.225 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.225 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.225 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for peered", 2024-04-26T01:08:10.225 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.226 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.226 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.226 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 378 2024-04-26T01:08:10.226 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.226 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.226 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.226 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.226 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436792+0000", 2024-04-26T01:08:10.226 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.226 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.226 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.226 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.226 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436792+0000", 2024-04-26T01:08:10.227 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.227 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.227 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436793+0000", 2024-04-26T01:08:10.227 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.92e-07 2024-04-26T01:08:10.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.227 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.227 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436798+0000", 2024-04-26T01:08:10.227 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3890000000000004e-06 2024-04-26T01:08:10.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.228 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.228 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436812+0000", 2024-04-26T01:08:10.228 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4243999999999999e-05 2024-04-26T01:08:10.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.228 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.228 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.228 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436820+0000", 2024-04-26T01:08:10.228 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.9389999999999997e-06 2024-04-26T01:08:10.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.228 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.228 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.228 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436872+0000", 2024-04-26T01:08:10.228 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1465000000000003e-05 2024-04-26T01:08:10.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.229 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.229 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:10.229 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436872+0000", 2024-04-26T01:08:10.229 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1999999999999998e-07 2024-04-26T01:08:10.229 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.229 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.229 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.229 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.229 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.229 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:379 3.7s0 3.86a9bb67 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.229 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.436838+0000", 2024-04-26T01:08:10.229 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.78653149700000002, 2024-04-26T01:08:10.230 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.78660843599999997, 2024-04-26T01:08:10.230 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.230 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.230 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for peered", 2024-04-26T01:08:10.230 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.230 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.230 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.230 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 379 2024-04-26T01:08:10.230 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.230 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.230 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.230 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.230 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436838+0000", 2024-04-26T01:08:10.230 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.231 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.231 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.231 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436838+0000", 2024-04-26T01:08:10.231 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.231 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.231 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.231 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436839+0000", 2024-04-26T01:08:10.231 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5499999999999998e-07 2024-04-26T01:08:10.231 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.231 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.231 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436843+0000", 2024-04-26T01:08:10.232 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0300000000000004e-06 2024-04-26T01:08:10.232 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.232 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.232 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436852+0000", 2024-04-26T01:08:10.232 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5169999999999999e-06 2024-04-26T01:08:10.232 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.232 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.232 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436858+0000", 2024-04-26T01:08:10.232 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2779999999999999e-06 2024-04-26T01:08:10.232 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.232 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.232 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436900+0000", 2024-04-26T01:08:10.233 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1695e-05 2024-04-26T01:08:10.233 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.233 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.233 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:10.233 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436900+0000", 2024-04-26T01:08:10.233 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.92e-07 2024-04-26T01:08:10.233 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.233 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.233 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.233 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.233 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.233 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:395 3.1s0 3.e9710d51 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.233 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.498664+0000", 2024-04-26T01:08:10.233 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.72470623300000003, 2024-04-26T01:08:10.234 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.72482271799999998, 2024-04-26T01:08:10.234 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.234 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.234 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for peered", 2024-04-26T01:08:10.234 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.234 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.234 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.234 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 395 2024-04-26T01:08:10.234 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.234 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.234 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.234 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.235 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.498664+0000", 2024-04-26T01:08:10.235 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.235 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.235 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.235 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.235 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.498664+0000", 2024-04-26T01:08:10.235 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.235 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.235 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.235 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.235 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.498667+0000", 2024-04-26T01:08:10.235 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4539999999999999e-06 2024-04-26T01:08:10.235 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.235 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.236 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.236 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.499212+0000", 2024-04-26T01:08:10.236 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00054489100000000004 2024-04-26T01:08:10.236 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.236 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.236 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.236 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.499243+0000", 2024-04-26T01:08:10.236 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1368999999999997e-05 2024-04-26T01:08:10.236 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.236 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.236 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.236 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.499259+0000", 2024-04-26T01:08:10.236 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5803000000000001e-05 2024-04-26T01:08:10.236 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.237 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.237 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.499358+0000", 2024-04-26T01:08:10.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8750000000000002e-05 2024-04-26T01:08:10.237 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.237 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:10.237 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.499362+0000", 2024-04-26T01:08:10.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4689999999999999e-06 2024-04-26T01:08:10.237 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.237 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.238 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.238 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.238 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.238 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:396 3.1s0 3.e9710d51 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.238 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.502111+0000", 2024-04-26T01:08:10.238 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.72125854899999997, 2024-04-26T01:08:10.238 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.72141246999999997, 2024-04-26T01:08:10.238 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.238 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.238 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for peered", 2024-04-26T01:08:10.238 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.238 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.239 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.239 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 396 2024-04-26T01:08:10.239 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.239 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.239 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.239 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.239 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.502111+0000", 2024-04-26T01:08:10.239 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.239 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.239 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.239 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.239 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.502111+0000", 2024-04-26T01:08:10.239 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.240 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.240 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.240 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.240 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.502113+0000", 2024-04-26T01:08:10.240 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.229e-06 2024-04-26T01:08:10.240 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.240 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.240 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.240 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.502785+0000", 2024-04-26T01:08:10.240 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00067241000000000002 2024-04-26T01:08:10.240 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.240 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.241 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.241 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.502805+0000", 2024-04-26T01:08:10.241 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9850000000000001e-05 2024-04-26T01:08:10.241 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.241 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.241 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.241 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.502815+0000", 2024-04-26T01:08:10.241 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0087e-05 2024-04-26T01:08:10.241 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.241 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.241 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.241 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.502899+0000", 2024-04-26T01:08:10.242 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4279999999999999e-05 2024-04-26T01:08:10.242 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.242 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:10.242 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.502904+0000", 2024-04-26T01:08:10.242 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8189999999999999e-06 2024-04-26T01:08:10.242 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.242 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.242 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.242 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.242 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:397 3.1s0 3.e9710d51 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.242 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.502950+0000", 2024-04-26T01:08:10.242 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.72041974600000003, 2024-04-26T01:08:10.243 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.72061152399999995, 2024-04-26T01:08:10.243 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.243 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.243 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for peered", 2024-04-26T01:08:10.243 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.243 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.243 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.243 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 397 2024-04-26T01:08:10.243 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.243 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.243 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.243 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.243 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.502950+0000", 2024-04-26T01:08:10.244 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.244 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.244 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.244 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.244 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.502950+0000", 2024-04-26T01:08:10.244 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.244 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.244 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.244 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.244 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.502951+0000", 2024-04-26T01:08:10.244 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3209999999999999e-06 2024-04-26T01:08:10.244 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.244 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.245 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.245 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503092+0000", 2024-04-26T01:08:10.245 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000140168 2024-04-26T01:08:10.245 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.245 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.245 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.245 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503103+0000", 2024-04-26T01:08:10.245 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1073999999999999e-05 2024-04-26T01:08:10.245 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.245 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.245 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.245 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503108+0000", 2024-04-26T01:08:10.245 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6640000000000003e-06 2024-04-26T01:08:10.246 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.246 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.246 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.246 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503158+0000", 2024-04-26T01:08:10.246 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9555000000000002e-05 2024-04-26T01:08:10.246 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.246 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.246 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:10.246 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503160+0000", 2024-04-26T01:08:10.246 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8309999999999999e-06 2024-04-26T01:08:10.246 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.246 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.246 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.246 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.247 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.247 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:398 3.1s0 3.e9710d51 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.247 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.503123+0000", 2024-04-26T01:08:10.247 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.72024704799999995, 2024-04-26T01:08:10.247 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.72047573200000004, 2024-04-26T01:08:10.247 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.247 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.247 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for peered", 2024-04-26T01:08:10.247 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.247 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.247 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.247 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 398 2024-04-26T01:08:10.247 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.247 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.248 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.248 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503123+0000", 2024-04-26T01:08:10.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.248 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.248 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.248 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503123+0000", 2024-04-26T01:08:10.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.248 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.248 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.248 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503124+0000", 2024-04-26T01:08:10.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1500000000000003e-07 2024-04-26T01:08:10.249 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.249 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.249 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.249 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503128+0000", 2024-04-26T01:08:10.249 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7280000000000003e-06 2024-04-26T01:08:10.249 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.249 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.249 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.249 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503136+0000", 2024-04-26T01:08:10.249 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5530000000000004e-06 2024-04-26T01:08:10.249 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.249 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.249 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.250 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503142+0000", 2024-04-26T01:08:10.250 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3280000000000002e-06 2024-04-26T01:08:10.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.250 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.250 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503192+0000", 2024-04-26T01:08:10.250 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.0059000000000003e-05 2024-04-26T01:08:10.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.250 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:10.250 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503193+0000", 2024-04-26T01:08:10.250 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1930000000000001e-06 2024-04-26T01:08:10.250 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.251 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.251 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.251 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:399 3.1s0 3.e9710d51 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.251 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.503152+0000", 2024-04-26T01:08:10.251 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.720218264, 2024-04-26T01:08:10.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.720484707, 2024-04-26T01:08:10.251 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.251 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.251 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for peered", 2024-04-26T01:08:10.251 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.251 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.252 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.252 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 399 2024-04-26T01:08:10.252 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.252 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.252 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.252 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.252 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503152+0000", 2024-04-26T01:08:10.252 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.252 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.252 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.252 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.252 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503152+0000", 2024-04-26T01:08:10.253 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.253 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.253 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503152+0000", 2024-04-26T01:08:10.253 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0800000000000004e-07 2024-04-26T01:08:10.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.253 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.253 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503155+0000", 2024-04-26T01:08:10.253 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5469999999999999e-06 2024-04-26T01:08:10.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503162+0000", 2024-04-26T01:08:10.254 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0829999999999998e-06 2024-04-26T01:08:10.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503167+0000", 2024-04-26T01:08:10.254 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5029999999999997e-06 2024-04-26T01:08:10.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503208+0000", 2024-04-26T01:08:10.255 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0295e-05 2024-04-26T01:08:10.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.255 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.255 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:10.255 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.503208+0000", 2024-04-26T01:08:10.255 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6300000000000001e-07 2024-04-26T01:08:10.255 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.255 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.255 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.255 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.255 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:400 3.1s0 3.5e9741b1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.255 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.505022+0000", 2024-04-26T01:08:10.255 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.71834759299999995, 2024-04-26T01:08:10.256 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.71865276300000003, 2024-04-26T01:08:10.256 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.256 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.256 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for peered", 2024-04-26T01:08:10.256 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.256 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.256 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.256 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 400 2024-04-26T01:08:10.256 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.256 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.256 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.256 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.256 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.505022+0000", 2024-04-26T01:08:10.256 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.257 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.257 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.257 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.505022+0000", 2024-04-26T01:08:10.257 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.257 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.257 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.257 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.505023+0000", 2024-04-26T01:08:10.257 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1900000000000001e-07 2024-04-26T01:08:10.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.257 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.257 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.257 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.505145+0000", 2024-04-26T01:08:10.257 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000122069 2024-04-26T01:08:10.258 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.258 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.258 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.258 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.505160+0000", 2024-04-26T01:08:10.258 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4338000000000001e-05 2024-04-26T01:08:10.258 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.258 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.258 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.258 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.505168+0000", 2024-04-26T01:08:10.258 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3310000000000004e-06 2024-04-26T01:08:10.258 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.258 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.258 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.259 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.505235+0000", 2024-04-26T01:08:10.259 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6814000000000001e-05 2024-04-26T01:08:10.259 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.259 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.259 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:10.259 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.505237+0000", 2024-04-26T01:08:10.259 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.238e-06 2024-04-26T01:08:10.259 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.259 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.259 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.259 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.259 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.259 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:354 3.1s0 3.80e8e111 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.259 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.404457+0000", 2024-04-26T01:08:10.260 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.81891312699999996, 2024-04-26T01:08:10.260 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.81925588999999999, 2024-04-26T01:08:10.260 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.260 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.260 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for peered", 2024-04-26T01:08:10.260 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.260 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.260 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.260 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 354 2024-04-26T01:08:10.260 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.260 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.260 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.260 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.260 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.404457+0000", 2024-04-26T01:08:10.261 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.261 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.261 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.261 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.261 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.404457+0000", 2024-04-26T01:08:10.261 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.261 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.261 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.261 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.261 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.404459+0000", 2024-04-26T01:08:10.261 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3479999999999998e-06 2024-04-26T01:08:10.261 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.261 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.261 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.262 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.405340+0000", 2024-04-26T01:08:10.262 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00088126100000000005 2024-04-26T01:08:10.262 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.262 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.262 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.262 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.405372+0000", 2024-04-26T01:08:10.262 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1912000000000002e-05 2024-04-26T01:08:10.262 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.262 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.262 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.262 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.405387+0000", 2024-04-26T01:08:10.262 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4426999999999999e-05 2024-04-26T01:08:10.262 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.262 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.263 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.263 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.405478+0000", 2024-04-26T01:08:10.263 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1415000000000005e-05 2024-04-26T01:08:10.263 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.263 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.263 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:10.263 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.405483+0000", 2024-04-26T01:08:10.263 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4360000000000004e-06 2024-04-26T01:08:10.263 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.263 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.263 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.263 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.263 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.263 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:355 3.1s0 3.80e8e111 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.264 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.408199+0000", 2024-04-26T01:08:10.264 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.81517065099999997, 2024-04-26T01:08:10.264 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.81555070200000002, 2024-04-26T01:08:10.264 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.264 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.264 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for peered", 2024-04-26T01:08:10.264 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.264 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.264 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.264 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 355 2024-04-26T01:08:10.264 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.264 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.264 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.264 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.265 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.408199+0000", 2024-04-26T01:08:10.265 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.265 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.265 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.265 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.265 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.408199+0000", 2024-04-26T01:08:10.265 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.265 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.265 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.265 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.265 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.408201+0000", 2024-04-26T01:08:10.265 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3740000000000001e-06 2024-04-26T01:08:10.266 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.266 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.266 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.266 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409114+0000", 2024-04-26T01:08:10.266 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00091339399999999999 2024-04-26T01:08:10.266 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.266 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.266 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.266 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409128+0000", 2024-04-26T01:08:10.266 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4100000000000001e-05 2024-04-26T01:08:10.266 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.266 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.266 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.267 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409141+0000", 2024-04-26T01:08:10.267 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2689e-05 2024-04-26T01:08:10.267 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.267 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.267 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.267 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409211+0000", 2024-04-26T01:08:10.267 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0522e-05 2024-04-26T01:08:10.267 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.267 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.267 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:10.267 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409216+0000", 2024-04-26T01:08:10.267 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1159999999999998e-06 2024-04-26T01:08:10.267 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.268 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.268 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.268 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.268 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.268 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:356 3.1s0 3.80e8e111 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.268 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.409164+0000", 2024-04-26T01:08:10.268 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.814206031, 2024-04-26T01:08:10.268 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.81462289200000004, 2024-04-26T01:08:10.268 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.268 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.268 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for peered", 2024-04-26T01:08:10.268 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.268 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.268 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.269 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 356 2024-04-26T01:08:10.269 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.269 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.269 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.269 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.269 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409164+0000", 2024-04-26T01:08:10.269 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.269 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.269 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.269 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.269 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409164+0000", 2024-04-26T01:08:10.269 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.270 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.270 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.270 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.270 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409165+0000", 2024-04-26T01:08:10.270 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3090000000000001e-06 2024-04-26T01:08:10.270 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.270 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.270 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.270 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409226+0000", 2024-04-26T01:08:10.270 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1230999999999997e-05 2024-04-26T01:08:10.270 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.270 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.271 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.271 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409241+0000", 2024-04-26T01:08:10.271 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4789e-05 2024-04-26T01:08:10.271 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.271 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.271 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.271 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409251+0000", 2024-04-26T01:08:10.271 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0122000000000001e-05 2024-04-26T01:08:10.271 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.271 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.271 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.271 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409315+0000", 2024-04-26T01:08:10.271 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4023999999999996e-05 2024-04-26T01:08:10.272 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.272 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.272 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:10.272 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409318+0000", 2024-04-26T01:08:10.272 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9679999999999998e-06 2024-04-26T01:08:10.272 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.272 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.272 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.272 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.272 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.272 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:357 3.1s0 3.80e8e111 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.272 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.409271+0000", 2024-04-26T01:08:10.272 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.81409928300000001, 2024-04-26T01:08:10.272 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.814552889, 2024-04-26T01:08:10.272 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.273 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.273 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for peered", 2024-04-26T01:08:10.273 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.273 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.273 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.273 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 357 2024-04-26T01:08:10.273 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.273 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.273 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.273 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.273 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409271+0000", 2024-04-26T01:08:10.273 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.273 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.273 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.274 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.274 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409271+0000", 2024-04-26T01:08:10.274 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.274 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.274 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.274 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.274 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409271+0000", 2024-04-26T01:08:10.274 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7000000000000004e-07 2024-04-26T01:08:10.274 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.274 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.274 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.274 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409276+0000", 2024-04-26T01:08:10.274 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9350000000000002e-06 2024-04-26T01:08:10.274 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.275 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.275 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.275 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409286+0000", 2024-04-26T01:08:10.275 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9650000000000003e-06 2024-04-26T01:08:10.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.275 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.275 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.275 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409294+0000", 2024-04-26T01:08:10.275 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8970000000000008e-06 2024-04-26T01:08:10.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.275 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.275 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.275 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409580+0000", 2024-04-26T01:08:10.275 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00028564699999999998 2024-04-26T01:08:10.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409582+0000", 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2790000000000001e-06 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:358 3.1s0 3.80e8e111 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.409534+0000", 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.81383591399999999, 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.814330577, 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.276 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for peered", 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 358 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409534+0000", 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409534+0000", 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.277 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.278 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.278 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.278 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409535+0000", 2024-04-26T01:08:10.278 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9800000000000002e-07 2024-04-26T01:08:10.278 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.278 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.278 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.278 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409540+0000", 2024-04-26T01:08:10.278 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4519999999999996e-06 2024-04-26T01:08:10.278 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.278 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.278 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.278 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409556+0000", 2024-04-26T01:08:10.278 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6175e-05 2024-04-26T01:08:10.279 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.279 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.279 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.279 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409566+0000", 2024-04-26T01:08:10.279 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.5670000000000001e-06 2024-04-26T01:08:10.279 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.279 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.279 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.279 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409615+0000", 2024-04-26T01:08:10.279 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9379999999999998e-05 2024-04-26T01:08:10.279 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.279 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.279 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:10.279 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.409616+0000", 2024-04-26T01:08:10.279 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.16e-07 2024-04-26T01:08:10.280 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.280 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.280 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.280 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.280 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.280 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:377 3.7s0 3.86a9bb67 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:10.280 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.436745+0000", 2024-04-26T01:08:10.280 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.78662495200000004, 2024-04-26T01:08:10.280 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.78715767599999997, 2024-04-26T01:08:10.280 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.280 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.280 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for peered", 2024-04-26T01:08:10.280 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.280 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.281 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.281 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 377 2024-04-26T01:08:10.281 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.281 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.281 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.281 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.281 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436745+0000", 2024-04-26T01:08:10.281 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.281 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.281 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.281 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.281 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436745+0000", 2024-04-26T01:08:10.281 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.281 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.282 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.282 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.282 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436746+0000", 2024-04-26T01:08:10.282 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1319999999999999e-06 2024-04-26T01:08:10.282 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.282 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.282 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.282 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436750+0000", 2024-04-26T01:08:10.282 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.985e-06 2024-04-26T01:08:10.282 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.282 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.282 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.282 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436763+0000", 2024-04-26T01:08:10.282 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2574e-05 2024-04-26T01:08:10.282 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.283 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.283 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.283 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436771+0000", 2024-04-26T01:08:10.283 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4859999999999997e-06 2024-04-26T01:08:10.283 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.283 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.283 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.283 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436840+0000", 2024-04-26T01:08:10.283 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9023999999999995e-05 2024-04-26T01:08:10.283 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.283 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.283 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:10.283 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436843+0000", 2024-04-26T01:08:10.284 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0290000000000001e-06 2024-04-26T01:08:10.284 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.284 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.284 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.284 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.284 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-26T01:08:10.284 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 14 2024-04-26T01:08:10.284 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:10.285 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:08:10.306 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:10.305+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:10.394 INFO:teuthology.orchestra.run.smithi008.stderr:marked out osd.11. 2024-04-26T01:08:10.400 INFO:tasks.rados.rados.0.smithi164.stdout:158: done (11 left) 2024-04-26T01:08:10.400 INFO:tasks.rados.rados.0.smithi164.stdout:159: done (10 left) 2024-04-26T01:08:10.401 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 18 current snap is 6 2024-04-26T01:08:10.401 INFO:tasks.rados.rados.0.smithi164.stdout:160: seq_num 84 ranges {294912=655360,950272=638976,1589248=131072} 2024-04-26T01:08:10.406 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:10.404+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:10.408 INFO:tasks.rados.rados.0.smithi164.stdout:160: writing smithi16428812-18 from 294912 to 950272 tid 1 2024-04-26T01:08:10.413 INFO:tasks.rados.rados.0.smithi164.stdout:160: writing smithi16428812-18 from 950272 to 1589248 tid 2 2024-04-26T01:08:10.414 INFO:tasks.rados.rados.0.smithi164.stdout:160: writing smithi16428812-18 from 1589248 to 1720320 tid 3 2024-04-26T01:08:10.414 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 7 current snap is 6 2024-04-26T01:08:10.414 INFO:tasks.rados.rados.0.smithi164.stdout:161: seq_num 85 ranges {114688=671744,786432=770048} 2024-04-26T01:08:10.419 INFO:tasks.rados.rados.0.smithi164.stdout:161: writing smithi16428812-7 from 114688 to 786432 tid 1 2024-04-26T01:08:10.425 INFO:tasks.rados.rados.0.smithi164.stdout:161: writing smithi16428812-7 from 786432 to 1556480 tid 2 2024-04-26T01:08:10.425 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 43 current snap is 6 2024-04-26T01:08:10.425 INFO:tasks.rados.rados.0.smithi164.stdout:162: seq_num 86 ranges {720896=688128,1409024=540672,1949696=229376} 2024-04-26T01:08:10.426 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.11 2024-04-26T01:08:10.427 INFO:tasks.thrashosds.thrasher:remote for osd 11 is ubuntu@smithi164.front.sepia.ceph.com 2024-04-26T01:08:10.427 DEBUG:teuthology.orchestra.run.smithi164:> type ceph-objectstore-tool 2024-04-26T01:08:10.430 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:10.430 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:10.430 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-04-26T01:08:10.430 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-04-26T01:08:10.430 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:10.432 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-26T01:08:10.432 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-11 --journal-path /var/lib/ceph/osd/ceph-11/journal --op list-pgs 2024-04-26T01:08:10.435 INFO:tasks.rados.rados.0.smithi164.stdout:162: writing smithi16428812-43 from 720896 to 1409024 tid 1 2024-04-26T01:08:10.438 INFO:tasks.rados.rados.0.smithi164.stdout:162: writing smithi16428812-43 from 1409024 to 1949696 tid 2 2024-04-26T01:08:10.440 INFO:tasks.rados.rados.0.smithi164.stdout:162: writing smithi16428812-43 from 1949696 to 2179072 tid 3 2024-04-26T01:08:10.441 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 37 current snap is 6 2024-04-26T01:08:10.441 INFO:tasks.rados.rados.0.smithi164.stdout:163: seq_num 87 ranges {0=704512,704512=688128,1392640=81920} 2024-04-26T01:08:10.443 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:08:10.446 INFO:tasks.rados.rados.0.smithi164.stdout:163: writing smithi16428812-37 from 0 to 704512 tid 1 2024-04-26T01:08:10.452 INFO:tasks.rados.rados.0.smithi164.stdout:163: writing smithi16428812-37 from 704512 to 1392640 tid 2 2024-04-26T01:08:10.453 INFO:tasks.rados.rados.0.smithi164.stdout:163: writing smithi16428812-37 from 1392640 to 1474560 tid 3 2024-04-26T01:08:10.453 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 21 current snap is 6 2024-04-26T01:08:10.453 INFO:tasks.rados.rados.0.smithi164.stdout:162: finishing write tid 1 to smithi16428812-43 2024-04-26T01:08:10.453 INFO:tasks.rados.rados.0.smithi164.stdout:162: finishing write tid 2 to smithi16428812-43 2024-04-26T01:08:10.453 INFO:tasks.rados.rados.0.smithi164.stdout:164: seq_num 88 ranges {1228800=720896,1949696=442368,2392064=327680} 2024-04-26T01:08:10.466 INFO:tasks.rados.rados.0.smithi164.stdout:164: writing smithi16428812-21 from 1228800 to 1949696 tid 1 2024-04-26T01:08:10.468 INFO:tasks.rados.rados.0.smithi164.stdout:164: writing smithi16428812-21 from 1949696 to 2392064 tid 2 2024-04-26T01:08:10.471 INFO:tasks.rados.rados.0.smithi164.stdout:164: writing smithi16428812-21 from 2392064 to 2719744 tid 3 2024-04-26T01:08:10.471 INFO:tasks.rados.rados.0.smithi164.stdout:165: read oid 40 snap -1 2024-04-26T01:08:10.471 INFO:tasks.rados.rados.0.smithi164.stdout:165: expect (ObjNum 39 snap 0 seq_num 39) 2024-04-26T01:08:10.471 INFO:tasks.rados.rados.0.smithi164.stdout: waiting on 16 2024-04-26T01:08:10.472 INFO:tasks.rados.rados.0.smithi164.stdout:162: finishing write tid 3 to smithi16428812-43 2024-04-26T01:08:10.472 INFO:tasks.rados.rados.0.smithi164.stdout:162: finishing write tid 4 to smithi16428812-43 2024-04-26T01:08:10.472 INFO:tasks.rados.rados.0.smithi164.stdout:163: finishing write tid 1 to smithi16428812-37 2024-04-26T01:08:10.472 INFO:tasks.rados.rados.0.smithi164.stdout:162: finishing write tid 5 to smithi16428812-43 2024-04-26T01:08:10.472 INFO:tasks.rados.rados.0.smithi164.stdout:162: oid 43 updating version 0 to 34 2024-04-26T01:08:10.472 INFO:tasks.rados.rados.0.smithi164.stdout:162: oid 43 updating version 34 to 35 2024-04-26T01:08:10.472 INFO:tasks.rados.rados.0.smithi164.stdout:162: oid 43 version 35 is already newer than 32 2024-04-26T01:08:10.472 INFO:tasks.rados.rados.0.smithi164.stdout:162: oid 43 version 35 is already newer than 33 2024-04-26T01:08:10.472 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 43 v 35 (ObjNum 86 snap 6 seq_num 86) dirty exists 2024-04-26T01:08:10.472 INFO:tasks.rados.rados.0.smithi164.stdout:162: left oid 43 (ObjNum 86 snap 6 seq_num 86) 2024-04-26T01:08:10.472 INFO:tasks.rados.rados.0.smithi164.stdout:163: finishing write tid 2 to smithi16428812-37 2024-04-26T01:08:10.472 INFO:tasks.rados.rados.0.smithi164.stdout:163: finishing write tid 3 to smithi16428812-37 2024-04-26T01:08:10.472 INFO:tasks.rados.rados.0.smithi164.stdout:162: done (15 left) 2024-04-26T01:08:10.472 INFO:tasks.rados.rados.0.smithi164.stdout:166: read oid 27 snap -1 2024-04-26T01:08:10.472 INFO:tasks.rados.rados.0.smithi164.stdout:166: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-26T01:08:10.473 INFO:tasks.rados.rados.0.smithi164.stdout: waiting on 16 2024-04-26T01:08:10.476 INFO:tasks.rados.rados.0.smithi164.stdout:165: expect (ObjNum 39 snap 0 seq_num 39) 2024-04-26T01:08:10.493 INFO:tasks.rados.rados.0.smithi164.stdout:165: done (15 left) 2024-04-26T01:08:10.493 INFO:tasks.rados.rados.0.smithi164.stdout:167: snap_create 2024-04-26T01:08:10.498 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:08:10.530 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:10.528+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:10.605 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:10.605 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:10.605 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:10.605 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:10.605 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.605 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:2 3.1s0 3:8bb39a4d:::smithi16428812-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:10.605 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.300456+0000", 2024-04-26T01:08:10.605 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 11.302407169, 2024-04-26T01:08:10.605 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018845095999999999, 2024-04-26T01:08:10.605 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.606 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.606 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:10.606 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.606 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.606 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.606 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 2 2024-04-26T01:08:10.606 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.606 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.606 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.606 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.606 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300456+0000", 2024-04-26T01:08:10.606 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.606 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.606 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.607 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.607 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300456+0000", 2024-04-26T01:08:10.607 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.607 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.607 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.607 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.607 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300457+0000", 2024-04-26T01:08:10.607 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0999999999999998e-07 2024-04-26T01:08:10.607 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.607 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.607 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.607 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300461+0000", 2024-04-26T01:08:10.607 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0330000000000002e-06 2024-04-26T01:08:10.607 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.608 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.608 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.608 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300468+0000", 2024-04-26T01:08:10.608 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.362e-06 2024-04-26T01:08:10.608 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.608 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.608 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.608 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300474+0000", 2024-04-26T01:08:10.608 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2920000000000003e-06 2024-04-26T01:08:10.608 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.608 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.608 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.608 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302477+0000", 2024-04-26T01:08:10.608 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0020036979999999999 2024-04-26T01:08:10.609 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.609 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.609 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.609 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.302595+0000", 2024-04-26T01:08:10.609 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011778900000000001 2024-04-26T01:08:10.609 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.609 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.609 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.609 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303203+0000", 2024-04-26T01:08:10.609 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00060815900000000004 2024-04-26T01:08:10.609 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.609 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.609 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.609 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307208+0000", 2024-04-26T01:08:10.609 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0040047399999999997 2024-04-26T01:08:10.610 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.610 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.610 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:10.610 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319228+0000", 2024-04-26T01:08:10.610 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012019556000000001 2024-04-26T01:08:10.610 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.610 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.610 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.610 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319301+0000", 2024-04-26T01:08:10.610 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3856999999999994e-05 2024-04-26T01:08:10.610 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.610 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.610 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.610 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.611 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.611 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:3 3.1s0 3:8bb39a4d:::smithi16428812-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:10.611 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.300507+0000", 2024-04-26T01:08:10.611 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 11.302356626, 2024-04-26T01:08:10.611 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.030795241000000001, 2024-04-26T01:08:10.611 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.611 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.611 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.611 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.611 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.611 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.611 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 3 2024-04-26T01:08:10.611 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.611 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.612 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.612 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.612 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300507+0000", 2024-04-26T01:08:10.612 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.612 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.612 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.612 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.612 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300507+0000", 2024-04-26T01:08:10.612 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.612 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.612 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.612 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300507+0000", 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2600000000000002e-07 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300510+0000", 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1940000000000001e-06 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300524+0000", 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4663e-05 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.613 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300529+0000", 2024-04-26T01:08:10.614 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8919999999999999e-06 2024-04-26T01:08:10.614 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.614 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.614 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.614 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303652+0000", 2024-04-26T01:08:10.614 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0031224500000000001 2024-04-26T01:08:10.614 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.614 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.614 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:10.614 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303744+0000", 2024-04-26T01:08:10.614 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2109999999999997e-05 2024-04-26T01:08:10.614 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.614 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.614 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319709+0000", 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015965231 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319786+0000", 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7449999999999999e-05 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.331302+0000", 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011515724999999999 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.615 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.616 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:19 3.7s0 3:e45e30e5:::smithi16428812-7:head [append 0~114688,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:10.616 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.300548+0000", 2024-04-26T01:08:10.616 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 11.302315177000001, 2024-04-26T01:08:10.616 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018601116000000001, 2024-04-26T01:08:10.616 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.616 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.616 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:10.616 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.616 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.616 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.616 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 19 2024-04-26T01:08:10.616 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.616 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.616 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.617 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.617 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300548+0000", 2024-04-26T01:08:10.617 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.617 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.617 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.617 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.617 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300548+0000", 2024-04-26T01:08:10.617 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.617 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.617 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.617 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.617 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300549+0000", 2024-04-26T01:08:10.617 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8999999999999997e-07 2024-04-26T01:08:10.618 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.618 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.618 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.618 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300708+0000", 2024-04-26T01:08:10.618 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015877 2024-04-26T01:08:10.618 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.618 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.618 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.618 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300716+0000", 2024-04-26T01:08:10.618 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7050000000000005e-06 2024-04-26T01:08:10.618 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.618 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.618 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.619 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300724+0000", 2024-04-26T01:08:10.619 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.9240000000000007e-06 2024-04-26T01:08:10.619 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.619 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.619 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.619 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300802+0000", 2024-04-26T01:08:10.619 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7812000000000006e-05 2024-04-26T01:08:10.619 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.619 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.619 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.619 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300958+0000", 2024-04-26T01:08:10.619 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015555000000000001 2024-04-26T01:08:10.619 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.619 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.620 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.620 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.301756+0000", 2024-04-26T01:08:10.620 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00079798900000000003 2024-04-26T01:08:10.620 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.620 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.620 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.620 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.305686+0000", 2024-04-26T01:08:10.620 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0039306769999999996 2024-04-26T01:08:10.620 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.620 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.620 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:10.620 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319067+0000", 2024-04-26T01:08:10.620 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013380906 2024-04-26T01:08:10.620 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.620 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.621 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.621 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319149+0000", 2024-04-26T01:08:10.621 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2293000000000006e-05 2024-04-26T01:08:10.621 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.621 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.621 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.621 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.621 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.621 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:21 3.7s0 3:e45e30e5:::smithi16428812-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:10.621 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.300810+0000", 2024-04-26T01:08:10.621 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 11.302053052, 2024-04-26T01:08:10.621 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.027297502000000001, 2024-04-26T01:08:10.621 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.622 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.622 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.622 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.622 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.622 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.622 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 21 2024-04-26T01:08:10.622 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.622 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.622 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.622 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.622 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300810+0000", 2024-04-26T01:08:10.622 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.622 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.622 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300810+0000", 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300811+0000", 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4600000000000005e-07 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300813+0000", 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4660000000000002e-06 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.623 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300819+0000", 2024-04-26T01:08:10.624 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9279999999999998e-06 2024-04-26T01:08:10.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.624 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.624 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.624 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300824+0000", 2024-04-26T01:08:10.624 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8690000000000003e-06 2024-04-26T01:08:10.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.624 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.624 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.624 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303216+0000", 2024-04-26T01:08:10.624 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002391743 2024-04-26T01:08:10.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.624 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.624 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:10.624 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303300+0000", 2024-04-26T01:08:10.625 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3857999999999996e-05 2024-04-26T01:08:10.625 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.625 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.625 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.625 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319712+0000", 2024-04-26T01:08:10.625 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.016412102000000001 2024-04-26T01:08:10.625 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.625 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.625 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.625 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319825+0000", 2024-04-26T01:08:10.625 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000113062 2024-04-26T01:08:10.625 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.625 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.625 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.625 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.328108+0000", 2024-04-26T01:08:10.626 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0082829280000000002 2024-04-26T01:08:10.626 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.626 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.626 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.626 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.626 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.626 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:38 3.7s0 3:ed7aa414:::smithi16428812-13:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.626 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.321672+0000", 2024-04-26T01:08:10.626 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 11.281191358999999, 2024-04-26T01:08:10.626 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.035147350000000001, 2024-04-26T01:08:10.626 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.626 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.626 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:10.626 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.627 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.627 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.627 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 38 2024-04-26T01:08:10.627 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.627 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.627 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.627 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.627 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321672+0000", 2024-04-26T01:08:10.627 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.627 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.627 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.627 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.627 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321672+0000", 2024-04-26T01:08:10.627 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321673+0000", 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0100000000000001e-06 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321680+0000", 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.024e-06 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321698+0000", 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8158000000000001e-05 2024-04-26T01:08:10.628 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321707+0000", 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6419999999999996e-06 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325010+0000", 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033028720000000001 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325101+0000", 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1613999999999994e-05 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.629 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.630 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.630 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325552+0000", 2024-04-26T01:08:10.630 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00045083800000000001 2024-04-26T01:08:10.630 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.630 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.630 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.630 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.328138+0000", 2024-04-26T01:08:10.630 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0025861159999999998 2024-04-26T01:08:10.630 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.630 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.630 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:10.630 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.356713+0000", 2024-04-26T01:08:10.630 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.028574609000000001 2024-04-26T01:08:10.631 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.631 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.631 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.631 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.356819+0000", 2024-04-26T01:08:10.631 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000106467 2024-04-26T01:08:10.631 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.631 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.631 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.631 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.631 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.631 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:39 3.7s0 3:ed7aa414:::smithi16428812-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.631 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.321721+0000", 2024-04-26T01:08:10.631 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 11.281142430999999, 2024-04-26T01:08:10.631 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.038875445000000002, 2024-04-26T01:08:10.631 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 39 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321721+0000", 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.632 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321721+0000", 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321722+0000", 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2999999999999999e-07 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321726+0000", 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7639999999999999e-06 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321736+0000", 2024-04-26T01:08:10.633 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0473e-05 2024-04-26T01:08:10.634 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.634 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.634 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.634 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321747+0000", 2024-04-26T01:08:10.634 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0981e-05 2024-04-26T01:08:10.634 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.634 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.634 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.634 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.326352+0000", 2024-04-26T01:08:10.634 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046046239999999999 2024-04-26T01:08:10.634 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.634 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.634 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:10.634 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.326448+0000", 2024-04-26T01:08:10.635 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6668000000000003e-05 2024-04-26T01:08:10.635 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.635 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.635 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.635 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.356889+0000", 2024-04-26T01:08:10.635 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.030441071 2024-04-26T01:08:10.635 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.635 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.635 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.635 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.357378+0000", 2024-04-26T01:08:10.635 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00048859800000000003 2024-04-26T01:08:10.635 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.635 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.635 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.360597+0000", 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032184359999999999 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:65 3.7s0 3:e163c084:::smithi16428812-22:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.364410+0000", 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 11.238453816, 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.081696361999999995, 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.637 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.637 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 65 2024-04-26T01:08:10.637 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.637 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.637 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.637 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.637 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364410+0000", 2024-04-26T01:08:10.637 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.637 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.637 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.637 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.637 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364410+0000", 2024-04-26T01:08:10.637 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.637 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364411+0000", 2024-04-26T01:08:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.102e-06 2024-04-26T01:08:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364416+0000", 2024-04-26T01:08:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8389999999999997e-06 2024-04-26T01:08:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364424+0000", 2024-04-26T01:08:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6100000000000006e-06 2024-04-26T01:08:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.639 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.639 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.639 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364432+0000", 2024-04-26T01:08:10.639 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5320000000000001e-06 2024-04-26T01:08:10.639 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.639 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.639 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.639 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.365925+0000", 2024-04-26T01:08:10.639 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001493592 2024-04-26T01:08:10.639 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.639 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.639 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.639 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366035+0000", 2024-04-26T01:08:10.639 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000109448 2024-04-26T01:08:10.640 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.640 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.640 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.640 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366490+0000", 2024-04-26T01:08:10.640 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000455427 2024-04-26T01:08:10.640 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.640 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.640 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.640 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.367513+0000", 2024-04-26T01:08:10.640 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001023014 2024-04-26T01:08:10.640 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.640 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.640 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:10.640 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446008+0000", 2024-04-26T01:08:10.640 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.078494871999999993 2024-04-26T01:08:10.641 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.641 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.641 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.641 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446106+0000", 2024-04-26T01:08:10.641 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7925999999999997e-05 2024-04-26T01:08:10.641 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.641 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.641 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.641 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.641 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.641 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:66 3.7s0 3:e163c084:::smithi16428812-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.641 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.364442+0000", 2024-04-26T01:08:10.641 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 11.238421288, 2024-04-26T01:08:10.641 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.088465429999999998, 2024-04-26T01:08:10.642 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.642 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.642 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.642 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.642 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.642 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.642 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 66 2024-04-26T01:08:10.642 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.642 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.642 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.642 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.642 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364442+0000", 2024-04-26T01:08:10.642 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.643 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.643 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.643 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.643 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364442+0000", 2024-04-26T01:08:10.643 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.643 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.643 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.643 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.643 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364443+0000", 2024-04-26T01:08:10.643 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7899999999999998e-07 2024-04-26T01:08:10.643 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.643 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.643 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.643 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364446+0000", 2024-04-26T01:08:10.643 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9119999999999998e-06 2024-04-26T01:08:10.644 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.644 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.644 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.644 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364454+0000", 2024-04-26T01:08:10.644 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0229999999999992e-06 2024-04-26T01:08:10.644 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.644 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.644 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.644 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364461+0000", 2024-04-26T01:08:10.644 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0110000000000003e-06 2024-04-26T01:08:10.644 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.644 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.644 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.644 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.367245+0000", 2024-04-26T01:08:10.645 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027847499999999999 2024-04-26T01:08:10.645 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.645 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.645 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:10.645 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.367332+0000", 2024-04-26T01:08:10.645 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6118000000000005e-05 2024-04-26T01:08:10.645 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.645 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.645 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.645 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446632+0000", 2024-04-26T01:08:10.645 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.079300689999999993 2024-04-26T01:08:10.645 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.645 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.645 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.645 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446716+0000", 2024-04-26T01:08:10.646 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3391000000000002e-05 2024-04-26T01:08:10.646 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.646 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.646 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.646 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.452908+0000", 2024-04-26T01:08:10.646 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0061919560000000002 2024-04-26T01:08:10.646 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.646 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.646 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.646 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.646 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.646 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:78 3.1s0 3:8f91d505:::smithi16428812-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.646 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.380021+0000", 2024-04-26T01:08:10.647 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 11.222842357999999, 2024-04-26T01:08:10.647 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.023701460000000001, 2024-04-26T01:08:10.647 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.647 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.647 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.647 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.647 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.647 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.647 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 78 2024-04-26T01:08:10.647 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.647 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.647 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.647 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.647 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380021+0000", 2024-04-26T01:08:10.648 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.648 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.648 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.648 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.648 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380021+0000", 2024-04-26T01:08:10.648 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.648 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.648 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.648 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.648 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380022+0000", 2024-04-26T01:08:10.648 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5700000000000002e-07 2024-04-26T01:08:10.648 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.648 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.649 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.649 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380025+0000", 2024-04-26T01:08:10.649 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0900000000000001e-06 2024-04-26T01:08:10.649 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.649 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.649 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.649 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380034+0000", 2024-04-26T01:08:10.649 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1039999999999996e-06 2024-04-26T01:08:10.649 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.649 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.649 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.649 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.380040+0000", 2024-04-26T01:08:10.649 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0599999999999996e-06 2024-04-26T01:08:10.649 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.650 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.650 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.650 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.383108+0000", 2024-04-26T01:08:10.650 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0030677109999999999 2024-04-26T01:08:10.650 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.650 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.650 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:10.650 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.383200+0000", 2024-04-26T01:08:10.650 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2689999999999995e-05 2024-04-26T01:08:10.650 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.650 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.650 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.650 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.389150+0000", 2024-04-26T01:08:10.650 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0059497919999999998 2024-04-26T01:08:10.650 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.651 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.651 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.651 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.389236+0000", 2024-04-26T01:08:10.651 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5784000000000004e-05 2024-04-26T01:08:10.651 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.651 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.651 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.651 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.403723+0000", 2024-04-26T01:08:10.651 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014486572 2024-04-26T01:08:10.651 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.651 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.651 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.651 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.651 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.652 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:93 3.1s0 3:8ab08e97:::smithi16428812-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.652 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.401436+0000", 2024-04-26T01:08:10.652 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 11.20142749, 2024-04-26T01:08:10.652 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018987277, 2024-04-26T01:08:10.652 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.652 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.652 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.652 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.652 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.652 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.652 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 93 2024-04-26T01:08:10.652 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.652 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.652 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.653 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.653 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401436+0000", 2024-04-26T01:08:10.653 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.653 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.653 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.653 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.653 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401436+0000", 2024-04-26T01:08:10.653 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.653 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.653 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.653 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.653 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401437+0000", 2024-04-26T01:08:10.653 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0200000000000001e-07 2024-04-26T01:08:10.653 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.653 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.654 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.654 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401440+0000", 2024-04-26T01:08:10.654 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6600000000000001e-06 2024-04-26T01:08:10.654 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.654 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.654 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.654 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401448+0000", 2024-04-26T01:08:10.654 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2110000000000001e-06 2024-04-26T01:08:10.654 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.654 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.654 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.654 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.401452+0000", 2024-04-26T01:08:10.654 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3460000000000004e-06 2024-04-26T01:08:10.654 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.654 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.403762+0000", 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0023097880000000001 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.403807+0000", 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.511e-05 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.417495+0000", 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013688117 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.655 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.656 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.417588+0000", 2024-04-26T01:08:10.656 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3445999999999999e-05 2024-04-26T01:08:10.656 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.656 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.656 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.656 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.420423+0000", 2024-04-26T01:08:10.656 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0028348969999999999 2024-04-26T01:08:10.656 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.656 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.656 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.656 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.656 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.656 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:102 3.1s0 3:88871701:::smithi16428812-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.656 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.413770+0000", 2024-04-26T01:08:10.656 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 11.189093593999999, 2024-04-26T01:08:10.657 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.035191759000000003, 2024-04-26T01:08:10.657 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.657 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.657 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.657 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.657 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.657 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.657 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 102 2024-04-26T01:08:10.657 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.657 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.657 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.657 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.657 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413770+0000", 2024-04-26T01:08:10.657 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.657 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.658 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.658 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.658 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413770+0000", 2024-04-26T01:08:10.658 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.658 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.658 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.658 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.658 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413770+0000", 2024-04-26T01:08:10.658 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4799999999999998e-07 2024-04-26T01:08:10.658 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.658 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.658 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.658 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413773+0000", 2024-04-26T01:08:10.659 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7499999999999999e-06 2024-04-26T01:08:10.659 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.659 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.659 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.659 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413782+0000", 2024-04-26T01:08:10.659 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7129999999999994e-06 2024-04-26T01:08:10.659 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.659 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.659 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.659 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413791+0000", 2024-04-26T01:08:10.659 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4280000000000002e-06 2024-04-26T01:08:10.659 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.659 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.659 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.659 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.416262+0000", 2024-04-26T01:08:10.660 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024706210000000001 2024-04-26T01:08:10.660 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.660 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.660 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:10.660 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.416358+0000", 2024-04-26T01:08:10.660 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.5775999999999999e-05 2024-04-26T01:08:10.660 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.660 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.660 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.660 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.419853+0000", 2024-04-26T01:08:10.660 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0034948790000000002 2024-04-26T01:08:10.660 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.660 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.660 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.661 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.419904+0000", 2024-04-26T01:08:10.661 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1206999999999999e-05 2024-04-26T01:08:10.661 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.661 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.661 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.661 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.448962+0000", 2024-04-26T01:08:10.661 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.029057837 2024-04-26T01:08:10.661 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.661 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.661 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.661 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.661 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.661 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:120 3.bs0 3:d5691d7a:::smithi16428812-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.661 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.443773+0000", 2024-04-26T01:08:10.661 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 11.159090403, 2024-04-26T01:08:10.662 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.020073483, 2024-04-26T01:08:10.662 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.662 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.662 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.662 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.662 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.662 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.662 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 120 2024-04-26T01:08:10.662 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.662 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.662 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.662 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.662 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.443773+0000", 2024-04-26T01:08:10.662 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.443773+0000", 2024-04-26T01:08:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.443774+0000", 2024-04-26T01:08:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2900000000000003e-07 2024-04-26T01:08:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.443777+0000", 2024-04-26T01:08:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1779999999999999e-06 2024-04-26T01:08:10.664 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.664 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.664 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.664 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.443786+0000", 2024-04-26T01:08:10.664 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1959999999999997e-06 2024-04-26T01:08:10.664 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.664 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.664 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.664 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.443794+0000", 2024-04-26T01:08:10.664 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.4800000000000004e-06 2024-04-26T01:08:10.664 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.664 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.664 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.664 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446188+0000", 2024-04-26T01:08:10.665 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002394864 2024-04-26T01:08:10.665 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.665 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.665 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:10.665 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446354+0000", 2024-04-26T01:08:10.665 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000165129 2024-04-26T01:08:10.665 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.665 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.665 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.665 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.454564+0000", 2024-04-26T01:08:10.665 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0082109120000000008 2024-04-26T01:08:10.665 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.665 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.665 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.665 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.454635+0000", 2024-04-26T01:08:10.666 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0937000000000002e-05 2024-04-26T01:08:10.666 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.666 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.666 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.666 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.463847+0000", 2024-04-26T01:08:10.666 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0092111580000000005 2024-04-26T01:08:10.666 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.666 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.666 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.666 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.666 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.666 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:144 3.bs0 3:d8d6d69b:::smithi16428812-48:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.666 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.477923+0000", 2024-04-26T01:08:10.666 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 11.124940580000001, 2024-04-26T01:08:10.667 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.022386774000000002, 2024-04-26T01:08:10.667 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.667 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.667 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:10.667 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.667 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.667 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.667 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 144 2024-04-26T01:08:10.667 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.667 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.667 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.667 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.667 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477923+0000", 2024-04-26T01:08:10.667 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.668 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.668 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.668 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.668 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477923+0000", 2024-04-26T01:08:10.668 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.668 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.668 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.668 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.668 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477924+0000", 2024-04-26T01:08:10.668 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0200000000000001e-07 2024-04-26T01:08:10.668 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.668 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.668 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.668 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477928+0000", 2024-04-26T01:08:10.669 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.053e-06 2024-04-26T01:08:10.669 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.669 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.669 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.669 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477936+0000", 2024-04-26T01:08:10.669 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5129999999999996e-06 2024-04-26T01:08:10.669 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.669 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.669 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.669 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477942+0000", 2024-04-26T01:08:10.669 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9569999999999999e-06 2024-04-26T01:08:10.669 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.669 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.669 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.670 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.479202+0000", 2024-04-26T01:08:10.670 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0012595600000000001 2024-04-26T01:08:10.670 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.670 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.670 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.670 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.479305+0000", 2024-04-26T01:08:10.670 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000103386 2024-04-26T01:08:10.670 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.670 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.670 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.670 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.479724+0000", 2024-04-26T01:08:10.670 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00041885799999999998 2024-04-26T01:08:10.670 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.670 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.671 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.671 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.480446+0000", 2024-04-26T01:08:10.671 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00072229900000000003 2024-04-26T01:08:10.671 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.671 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.671 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:10.671 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.500215+0000", 2024-04-26T01:08:10.671 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.019768626000000001 2024-04-26T01:08:10.671 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.671 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.671 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.671 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.500310+0000", 2024-04-26T01:08:10.671 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4820000000000004e-05 2024-04-26T01:08:10.671 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.671 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.672 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.672 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.672 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.672 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:145 3.bs0 3:d8d6d69b:::smithi16428812-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.672 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.477949+0000", 2024-04-26T01:08:10.672 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 11.12491462, 2024-04-26T01:08:10.672 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.030723740999999999, 2024-04-26T01:08:10.672 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.672 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.672 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.672 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.672 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.672 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.672 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 145 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477949+0000", 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477949+0000", 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477949+0000", 2024-04-26T01:08:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.5999999999999999e-07 2024-04-26T01:08:10.674 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.674 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.674 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.674 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477951+0000", 2024-04-26T01:08:10.674 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9929999999999998e-06 2024-04-26T01:08:10.674 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.674 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.674 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.674 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477955+0000", 2024-04-26T01:08:10.674 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2760000000000002e-06 2024-04-26T01:08:10.674 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.674 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.674 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.674 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477959+0000", 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7000000000000002e-06 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.480261+0000", 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0023015930000000002 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.480344+0000", 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3113000000000002e-05 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.501225+0000", 2024-04-26T01:08:10.675 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.020881591000000001 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.501312+0000", 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6468e-05 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.508673+0000", 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0073606469999999997 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.676 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:154 3.7s0 3:e8b1d757:::smithi16428812-42:head [copy-from ver 10] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:10.677 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.489221+0000", 2024-04-26T01:08:10.677 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 11.113642126, 2024-04-26T01:08:10.677 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018704502000000001, 2024-04-26T01:08:10.677 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.677 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.677 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:10.677 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.677 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.677 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.677 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154 2024-04-26T01:08:10.677 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.677 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.677 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.677 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.678 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489221+0000", 2024-04-26T01:08:10.678 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.678 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.678 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.678 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.678 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489221+0000", 2024-04-26T01:08:10.678 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.678 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.678 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.678 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.678 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489223+0000", 2024-04-26T01:08:10.678 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.454e-06 2024-04-26T01:08:10.678 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.678 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.678 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.679 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489228+0000", 2024-04-26T01:08:10.679 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3199999999999999e-06 2024-04-26T01:08:10.679 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.679 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.679 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.679 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489244+0000", 2024-04-26T01:08:10.679 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5532999999999998e-05 2024-04-26T01:08:10.679 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.679 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.679 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.679 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489253+0000", 2024-04-26T01:08:10.679 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2439999999999999e-06 2024-04-26T01:08:10.679 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.679 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.680 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.680 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489326+0000", 2024-04-26T01:08:10.680 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3314999999999999e-05 2024-04-26T01:08:10.680 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.680 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.680 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.680 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.489436+0000", 2024-04-26T01:08:10.680 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010927199999999999 2024-04-26T01:08:10.680 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.680 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.680 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.680 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.499328+0000", 2024-04-26T01:08:10.680 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0098929489999999998 2024-04-26T01:08:10.680 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.681 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.681 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.681 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.501425+0000", 2024-04-26T01:08:10.681 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0020967799999999999 2024-04-26T01:08:10.681 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.681 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.681 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:10.681 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.507827+0000", 2024-04-26T01:08:10.681 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0064014789999999998 2024-04-26T01:08:10.681 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.681 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.681 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.681 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.507926+0000", 2024-04-26T01:08:10.681 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9155999999999997e-05 2024-04-26T01:08:10.681 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.682 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.682 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.682 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.682 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.682 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.5s2 37/30 ECSubWrite(tid=22, reqid=client.4521.0:251, at_version=37'17, trim_to=0'0, roll_forward_to=36'13))", 2024-04-26T01:08:10.682 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.434740+0000", 2024-04-26T01:08:10.682 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 7.1681232990000003, 2024-04-26T01:08:10.682 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.47257335099999997, 2024-04-26T01:08:10.682 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.682 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.682 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:10.682 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.682 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.683 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.683 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434740+0000", 2024-04-26T01:08:10.683 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.683 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.683 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.683 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.683 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434740+0000", 2024-04-26T01:08:10.683 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.683 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.683 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.683 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.683 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434741+0000", 2024-04-26T01:08:10.683 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8800000000000002e-07 2024-04-26T01:08:10.683 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.683 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434764+0000", 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3499999999999999e-05 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906023+0000", 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.47125867799999999 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906041+0000", 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7887e-05 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.684 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.685 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906172+0000", 2024-04-26T01:08:10.685 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013065200000000001 2024-04-26T01:08:10.685 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.685 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.685 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.685 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906192+0000", 2024-04-26T01:08:10.685 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0363000000000001e-05 2024-04-26T01:08:10.685 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.685 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.685 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.685 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.907240+0000", 2024-04-26T01:08:10.685 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0010479 2024-04-26T01:08:10.685 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.685 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.685 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.686 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.907314+0000", 2024-04-26T01:08:10.686 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3583000000000003e-05 2024-04-26T01:08:10.686 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.686 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.686 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.686 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.686 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.686 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:267 3.1s0 3:8f91d505:::smithi16428812-26:head [copy-from ver 9] snapc 4=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:10.686 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.413377+0000", 2024-04-26T01:08:10.686 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 6.1894864050000002, 2024-04-26T01:08:10.686 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.033076248000000003, 2024-04-26T01:08:10.686 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.686 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.686 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:10.687 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.687 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.687 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.687 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 267 2024-04-26T01:08:10.687 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.687 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.687 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.687 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.687 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413377+0000", 2024-04-26T01:08:10.687 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.687 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.687 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.687 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.687 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413377+0000", 2024-04-26T01:08:10.687 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.688 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.688 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.688 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.688 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413382+0000", 2024-04-26T01:08:10.688 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5499999999999996e-06 2024-04-26T01:08:10.688 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.688 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.688 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.688 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413400+0000", 2024-04-26T01:08:10.688 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8646000000000001e-05 2024-04-26T01:08:10.688 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.688 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.688 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.688 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413453+0000", 2024-04-26T01:08:10.689 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2559999999999998e-05 2024-04-26T01:08:10.689 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.689 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.689 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.689 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413479+0000", 2024-04-26T01:08:10.689 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6210000000000001e-05 2024-04-26T01:08:10.689 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.689 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.689 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.689 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413657+0000", 2024-04-26T01:08:10.689 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017758800000000001 2024-04-26T01:08:10.689 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.689 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.689 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op must wait for map", 2024-04-26T01:08:10.690 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413721+0000", 2024-04-26T01:08:10.690 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4286000000000001e-05 2024-04-26T01:08:10.690 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.690 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.690 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.690 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.420637+0000", 2024-04-26T01:08:10.690 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0069159130000000001 2024-04-26T01:08:10.690 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.690 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.690 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.690 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.420830+0000", 2024-04-26T01:08:10.690 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000193492 2024-04-26T01:08:10.690 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.690 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.691 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.691 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.439350+0000", 2024-04-26T01:08:10.691 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018519566000000001 2024-04-26T01:08:10.691 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.691 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.691 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.691 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.442330+0000", 2024-04-26T01:08:10.691 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0029803 2024-04-26T01:08:10.691 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.691 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.691 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:10.691 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.446325+0000", 2024-04-26T01:08:10.691 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0039946699999999996 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.446453+0000", 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000128467 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:268 3.1s0 3:8f91d505:::smithi16428812-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.413543+0000", 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 6.1893207129999999, 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.028292085000000002, 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.692 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.693 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:10.693 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.693 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.693 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.693 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 268 2024-04-26T01:08:10.693 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.693 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.693 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.693 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.693 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413543+0000", 2024-04-26T01:08:10.693 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.693 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.693 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.693 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.694 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413543+0000", 2024-04-26T01:08:10.694 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.694 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.694 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.694 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.694 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413544+0000", 2024-04-26T01:08:10.694 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.59e-06 2024-04-26T01:08:10.694 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.694 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.694 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.694 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413556+0000", 2024-04-26T01:08:10.694 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1246999999999999e-05 2024-04-26T01:08:10.694 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.694 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.695 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.695 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413581+0000", 2024-04-26T01:08:10.695 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5111999999999999e-05 2024-04-26T01:08:10.695 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.695 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.695 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.695 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413599+0000", 2024-04-26T01:08:10.695 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8317000000000001e-05 2024-04-26T01:08:10.695 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.695 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.695 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.695 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413876+0000", 2024-04-26T01:08:10.695 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00027735100000000001 2024-04-26T01:08:10.695 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.695 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413893+0000", 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6917000000000001e-05 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.421170+0000", 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0072770140000000001 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for blocked object", 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.421226+0000", 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5999999999999999e-05 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.696 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.441576+0000", 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.020349750999999999 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.441701+0000", 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012496799999999999 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.441835+0000", 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013381800000000001 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.697 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.698 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:278 3.1s0 3:8d82e97a:::smithi16428812-25:head [rollback 2] snapc 4=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:10.698 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.414034+0000", 2024-04-26T01:08:10.698 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 6.1888299409999998, 2024-04-26T01:08:10.698 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.028063502000000001, 2024-04-26T01:08:10.698 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.698 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.698 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:10.698 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.698 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.698 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.698 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 278 2024-04-26T01:08:10.698 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.698 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.698 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.699 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.699 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414034+0000", 2024-04-26T01:08:10.699 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.699 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.699 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.699 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.699 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414034+0000", 2024-04-26T01:08:10.699 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.699 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.699 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.699 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.699 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414035+0000", 2024-04-26T01:08:10.699 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4500000000000001e-06 2024-04-26T01:08:10.699 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.700 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.700 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.700 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414043+0000", 2024-04-26T01:08:10.700 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7130000000000005e-06 2024-04-26T01:08:10.700 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.700 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.700 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.700 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414064+0000", 2024-04-26T01:08:10.700 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1169999999999999e-05 2024-04-26T01:08:10.700 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.700 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.700 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.700 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414211+0000", 2024-04-26T01:08:10.700 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014751599999999999 2024-04-26T01:08:10.701 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.701 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.701 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.701 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414358+0000", 2024-04-26T01:08:10.701 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014641799999999999 2024-04-26T01:08:10.701 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.701 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.701 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:10.701 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414375+0000", 2024-04-26T01:08:10.701 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6969999999999998e-05 2024-04-26T01:08:10.701 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.701 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.701 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.701 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.423720+0000", 2024-04-26T01:08:10.702 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0093450059999999995 2024-04-26T01:08:10.702 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.702 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.702 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.702 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.423968+0000", 2024-04-26T01:08:10.702 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000248151 2024-04-26T01:08:10.702 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.702 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.702 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.702 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.425217+0000", 2024-04-26T01:08:10.702 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0012493459999999999 2024-04-26T01:08:10.702 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.702 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.702 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.702 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.429803+0000", 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0045861840000000001 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.442013+0000", 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012209512 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.442097+0000", 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4066000000000005e-05 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.703 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:279 3.1s0 3:8d82e97a:::smithi16428812-25:head [append 0~0] snapc 4=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.414260+0000", 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 6.1886030270000001, 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.027997787, 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 279 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:10.704 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414260+0000", 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414260+0000", 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414262+0000", 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0099999999999998e-06 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414272+0000", 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1300000000000007e-06 2024-04-26T01:08:10.705 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.706 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.706 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:10.706 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414294+0000", 2024-04-26T01:08:10.706 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2484e-05 2024-04-26T01:08:10.706 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.706 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.706 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:10.706 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414311+0000", 2024-04-26T01:08:10.706 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6929999999999999e-05 2024-04-26T01:08:10.706 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.706 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.706 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.706 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414441+0000", 2024-04-26T01:08:10.706 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012956900000000001 2024-04-26T01:08:10.706 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.707 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.707 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:10.707 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414456+0000", 2024-04-26T01:08:10.707 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5273000000000001e-05 2024-04-26T01:08:10.707 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.707 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.707 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:10.707 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.426653+0000", 2024-04-26T01:08:10.707 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.01219716 2024-04-26T01:08:10.707 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.707 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.707 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:10.707 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.426952+0000", 2024-04-26T01:08:10.707 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00029896399999999998 2024-04-26T01:08:10.707 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.428812+0000", 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001859533 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.431969+0000", 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0031576170000000001 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.442211+0000", 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010241422999999999 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:10.708 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:10.709 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:10.709 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.442258+0000", 2024-04-26T01:08:10.709 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7694000000000001e-05 2024-04-26T01:08:10.709 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.709 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.709 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.709 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:10.709 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:10.709 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:10.711 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:08:10.711 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:10.711 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:10.712 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:10.705+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:10.714 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:10.714 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:10.743 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:10.743 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:10.771 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:10.771 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:10.798 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:10.798 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:10.811 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:10.808+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:10.828 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:10.828 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:10.845 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:10.845 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:10.845 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-04-26T01:08:10.845 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:10.856 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:10.856 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:10.860 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:08:10.887 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:10.887 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:10.911 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:10.911 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:10.911 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:10.912+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:10.937 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:10.937 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:10.963 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:10.963 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:10.964 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-26T01:08:10.986 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:10.987 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:11.012 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:11.012+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:11.013 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:11.013 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:11.013 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:11.013 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:11.031 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:08:11.034 INFO:tasks.rados.rados.0.smithi164.stdout:167: done (15 left) 2024-04-26T01:08:11.034 INFO:tasks.rados.rados.0.smithi164.stdout:168: delete oid 40 current snap is 7 2024-04-26T01:08:11.048 INFO:tasks.rados.rados.0.smithi164.stdout:163: finishing write tid 4 to smithi16428812-37 2024-04-26T01:08:11.048 INFO:tasks.rados.rados.0.smithi164.stdout:168: done (15 left) 2024-04-26T01:08:11.048 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 15 current snap is 7 2024-04-26T01:08:11.048 INFO:tasks.rados.rados.0.smithi164.stdout:169: seq_num 90 ranges {1638400=753664,2392064=720896,3112960=49152} 2024-04-26T01:08:11.059 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:11.060 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:11.062 INFO:tasks.rados.rados.0.smithi164.stdout:169: writing smithi16428812-15 from 1638400 to 2392064 tid 1 2024-04-26T01:08:11.066 INFO:tasks.rados.rados.0.smithi164.stdout:169: writing smithi16428812-15 from 2392064 to 3112960 tid 2 2024-04-26T01:08:11.068 INFO:tasks.rados.rados.0.smithi164.stdout:169: writing smithi16428812-15 from 3112960 to 3162112 tid 3 2024-04-26T01:08:11.069 INFO:tasks.rados.rados.0.smithi164.stdout: waiting on 16 2024-04-26T01:08:11.069 INFO:tasks.rados.rados.0.smithi164.stdout:163: finishing write tid 5 to smithi16428812-37 2024-04-26T01:08:11.069 INFO:tasks.rados.rados.0.smithi164.stdout:163: oid 37 updating version 0 to 11 2024-04-26T01:08:11.069 INFO:tasks.rados.rados.0.smithi164.stdout:163: oid 37 version 11 is already newer than 10 2024-04-26T01:08:11.069 INFO:tasks.rados.rados.0.smithi164.stdout:163: oid 37 updating version 11 to 12 2024-04-26T01:08:11.069 INFO:tasks.rados.rados.0.smithi164.stdout:163: oid 37 updating version 12 to 13 2024-04-26T01:08:11.069 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 37 v 13 (ObjNum 87 snap 6 seq_num 87) dirty exists 2024-04-26T01:08:11.069 INFO:tasks.rados.rados.0.smithi164.stdout:163: left oid 37 (ObjNum 87 snap 6 seq_num 87) 2024-04-26T01:08:11.069 INFO:tasks.rados.rados.0.smithi164.stdout:163: done (15 left) 2024-04-26T01:08:11.069 INFO:tasks.rados.rados.0.smithi164.stdout:170: rmattr oid 44 current snap is 7 2024-04-26T01:08:11.069 INFO:tasks.rados.rados.0.smithi164.stdout:170: done (15 left) 2024-04-26T01:08:11.069 INFO:tasks.rados.rados.0.smithi164.stdout:171: read oid 13 snap 5 2024-04-26T01:08:11.069 INFO:tasks.rados.rados.0.smithi164.stdout:171: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-26T01:08:11.069 INFO:tasks.rados.rados.0.smithi164.stdout: waiting on 16 2024-04-26T01:08:11.079 INFO:tasks.rados.rados.0.smithi164.stdout:169: finishing write tid 1 to smithi16428812-15 2024-04-26T01:08:11.084 INFO:tasks.rados.rados.0.smithi164.stdout:169: finishing write tid 2 to smithi16428812-15 2024-04-26T01:08:11.085 INFO:tasks.rados.rados.0.smithi164.stdout:149: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-26T01:08:11.094 INFO:tasks.rados.rados.0.smithi164.stdout:169: finishing write tid 3 to smithi16428812-15 2024-04-26T01:08:11.094 INFO:tasks.rados.rados.0.smithi164.stdout:169: finishing write tid 4 to smithi16428812-15 2024-04-26T01:08:11.094 INFO:tasks.rados.rados.0.smithi164.stdout:149: done (15 left) 2024-04-26T01:08:11.095 INFO:tasks.rados.rados.0.smithi164.stdout:172: copy_from oid 17 from oid 26 current snap is 7 2024-04-26T01:08:11.095 INFO:tasks.rados.rados.0.smithi164.stdout:142: finishing write tid 1 to smithi16428812-34 2024-04-26T01:08:11.095 INFO:tasks.rados.rados.0.smithi164.stdout:166: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-26T01:08:11.097 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:11.097 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:11.103 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T01:08:11.104 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T01:08:11.119 INFO:tasks.rados.rados.0.smithi164.stdout:169: finishing write tid 5 to smithi16428812-15 2024-04-26T01:08:11.119 INFO:tasks.rados.rados.0.smithi164.stdout:169: oid 15 updating version 0 to 19 2024-04-26T01:08:11.119 INFO:tasks.rados.rados.0.smithi164.stdout:169: oid 15 updating version 19 to 21 2024-04-26T01:08:11.119 INFO:tasks.rados.rados.0.smithi164.stdout:169: oid 15 updating version 21 to 22 2024-04-26T01:08:11.119 INFO:tasks.rados.rados.0.smithi164.stdout:169: oid 15 version 22 is already newer than 20 2024-04-26T01:08:11.119 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 15 v 22 (ObjNum 90 snap 7 seq_num 90) dirty exists 2024-04-26T01:08:11.119 INFO:tasks.rados.rados.0.smithi164.stdout:169: left oid 15 (ObjNum 90 snap 7 seq_num 90) 2024-04-26T01:08:11.119 INFO:tasks.rados.rados.0.smithi164.stdout:166: done (15 left) 2024-04-26T01:08:11.119 INFO:tasks.rados.rados.0.smithi164.stdout:169: done (14 left) 2024-04-26T01:08:11.119 INFO:tasks.rados.rados.0.smithi164.stdout:173: read oid 6 snap -1 2024-04-26T01:08:11.120 INFO:tasks.rados.rados.0.smithi164.stdout:173: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-26T01:08:11.120 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 42 current snap is 7 2024-04-26T01:08:11.120 INFO:tasks.rados.rados.0.smithi164.stdout:154: finishing write tid 1 to smithi16428812-31 2024-04-26T01:08:11.120 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 5 v 18 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-26T01:08:11.120 INFO:tasks.rados.rados.0.smithi164.stdout:174: seq_num 91 ranges {0=770048,770048=475136,1245184=294912} 2024-04-26T01:08:11.130 INFO:tasks.rados.rados.0.smithi164.stdout:174: writing smithi16428812-42 from 0 to 770048 tid 1 2024-04-26T01:08:11.136 INFO:tasks.rados.rados.0.smithi164.stdout:174: writing smithi16428812-42 from 770048 to 1245184 tid 2 2024-04-26T01:08:11.141 INFO:tasks.rados.rados.0.smithi164.stdout:174: writing smithi16428812-42 from 1245184 to 1540096 tid 3 2024-04-26T01:08:11.141 INFO:tasks.rados.rados.0.smithi164.stdout:144: done (15 left) 2024-04-26T01:08:11.141 INFO:tasks.rados.rados.0.smithi164.stdout:175: copy_from oid 46 from oid 8 current snap is 7 2024-04-26T01:08:11.141 INFO:tasks.rados.rados.0.smithi164.stdout: waiting on 16 2024-04-26T01:08:11.141 INFO:tasks.rados.rados.0.smithi164.stdout:154: finishing write tid 2 to smithi16428812-31 2024-04-26T01:08:11.141 INFO:tasks.rados.rados.0.smithi164.stdout:154: finishing write tid 3 to smithi16428812-31 2024-04-26T01:08:11.141 INFO:tasks.rados.rados.0.smithi164.stdout:154: finishing write tid 4 to smithi16428812-31 2024-04-26T01:08:11.141 INFO:tasks.rados.rados.0.smithi164.stdout:142: finishing write tid 2 to smithi16428812-34 2024-04-26T01:08:11.141 INFO:tasks.rados.rados.0.smithi164.stdout:151: expect (ObjNum 62 snap 2 seq_num 62) 2024-04-26T01:08:11.146 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:11.144+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:11.188 INFO:tasks.rados.rados.0.smithi164.stdout:164: finishing write tid 1 to smithi16428812-21 2024-04-26T01:08:11.188 INFO:tasks.rados.rados.0.smithi164.stdout:154: finishing write tid 5 to smithi16428812-31 2024-04-26T01:08:11.188 INFO:tasks.rados.rados.0.smithi164.stdout:154: oid 31 updating version 0 to 48 2024-04-26T01:08:11.188 INFO:tasks.rados.rados.0.smithi164.stdout:154: oid 31 version 48 is already newer than 47 2024-04-26T01:08:11.188 INFO:tasks.rados.rados.0.smithi164.stdout:154: oid 31 version 48 is already newer than 45 2024-04-26T01:08:11.188 INFO:tasks.rados.rados.0.smithi164.stdout:154: oid 31 version 48 is already newer than 46 2024-04-26T01:08:11.188 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 31 v 48 (ObjNum 83 snap 5 seq_num 83) dirty exists 2024-04-26T01:08:11.188 INFO:tasks.rados.rados.0.smithi164.stdout:154: left oid 31 (ObjNum 83 snap 5 seq_num 83) 2024-04-26T01:08:11.188 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 25 v 50 (ObjNum 72 snap 4 seq_num 72) dirty exists 2024-04-26T01:08:11.188 INFO:tasks.rados.rados.0.smithi164.stdout:164: finishing write tid 2 to smithi16428812-21 2024-04-26T01:08:11.188 INFO:tasks.rados.rados.0.smithi164.stdout:164: finishing write tid 3 to smithi16428812-21 2024-04-26T01:08:11.188 INFO:tasks.rados.rados.0.smithi164.stdout:142: finishing write tid 3 to smithi16428812-34 2024-04-26T01:08:11.189 INFO:tasks.rados.rados.0.smithi164.stdout:151: done (15 left) 2024-04-26T01:08:11.189 INFO:tasks.rados.rados.0.smithi164.stdout:154: done (14 left) 2024-04-26T01:08:11.189 INFO:tasks.rados.rados.0.smithi164.stdout:156: done (13 left) 2024-04-26T01:08:11.189 INFO:tasks.rados.rados.0.smithi164.stdout:176: rollback oid 27 current snap is 7 2024-04-26T01:08:11.189 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 27 to 6 2024-04-26T01:08:11.189 INFO:tasks.rados.rados.0.smithi164.stdout:164: finishing write tid 4 to smithi16428812-21 2024-04-26T01:08:11.189 INFO:tasks.rados.rados.0.smithi164.stdout:172: finishing copy_from racing read to smithi16428812-17 2024-04-26T01:08:11.189 INFO:tasks.rados.rados.0.smithi164.stdout:177: delete oid 50 current snap is 7 2024-04-26T01:08:11.189 INFO:tasks.rados.rados.0.smithi164.stdout:173: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-26T01:08:11.192 INFO:tasks.rados.rados.0.smithi164.stdout:164: finishing write tid 5 to smithi16428812-21 2024-04-26T01:08:11.192 INFO:tasks.rados.rados.0.smithi164.stdout:164: oid 21 updating version 0 to 34 2024-04-26T01:08:11.192 INFO:tasks.rados.rados.0.smithi164.stdout:164: oid 21 version 34 is already newer than 33 2024-04-26T01:08:11.192 INFO:tasks.rados.rados.0.smithi164.stdout:164: oid 21 updating version 34 to 36 2024-04-26T01:08:11.192 INFO:tasks.rados.rados.0.smithi164.stdout:164: oid 21 version 36 is already newer than 35 2024-04-26T01:08:11.192 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 21 v 36 (ObjNum 88 snap 6 seq_num 88) dirty exists 2024-04-26T01:08:11.193 INFO:tasks.rados.rados.0.smithi164.stdout:164: left oid 21 (ObjNum 88 snap 6 seq_num 88) 2024-04-26T01:08:11.193 INFO:tasks.rados.rados.0.smithi164.stdout:142: finishing write tid 4 to smithi16428812-34 2024-04-26T01:08:11.193 INFO:tasks.rados.rados.0.smithi164.stdout:153: finishing copy_from racing read to smithi16428812-36 2024-04-26T01:08:11.193 INFO:tasks.rados.rados.0.smithi164.stdout:160: finishing write tid 1 to smithi16428812-18 2024-04-26T01:08:11.193 INFO:tasks.rados.rados.0.smithi164.stdout:160: finishing write tid 2 to smithi16428812-18 2024-04-26T01:08:11.193 INFO:tasks.rados.rados.0.smithi164.stdout:160: finishing write tid 3 to smithi16428812-18 2024-04-26T01:08:11.193 INFO:tasks.rados.rados.0.smithi164.stdout:142: finishing write tid 5 to smithi16428812-34 2024-04-26T01:08:11.193 INFO:tasks.rados.rados.0.smithi164.stdout:142: oid 34 updating version 0 to 51 2024-04-26T01:08:11.193 INFO:tasks.rados.rados.0.smithi164.stdout:142: oid 34 version 51 is already newer than 44 2024-04-26T01:08:11.193 INFO:tasks.rados.rados.0.smithi164.stdout:142: oid 34 version 51 is already newer than 49 2024-04-26T01:08:11.193 INFO:tasks.rados.rados.0.smithi164.stdout:142: oid 34 updating version 51 to 52 2024-04-26T01:08:11.193 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 34 v 52 (ObjNum 78 snap 5 seq_num 78) dirty exists 2024-04-26T01:08:11.193 INFO:tasks.rados.rados.0.smithi164.stdout:142: left oid 34 (ObjNum 78 snap 5 seq_num 78) 2024-04-26T01:08:11.194 INFO:tasks.rados.rados.0.smithi164.stdout:150: finishing write tid 1 to smithi16428812-33 2024-04-26T01:08:11.194 INFO:tasks.rados.rados.0.smithi164.stdout:145: finishing copy_from racing read to smithi16428812-39 2024-04-26T01:08:11.194 INFO:tasks.rados.rados.0.smithi164.stdout:145: finishing copy_from to smithi16428812-39 2024-04-26T01:08:11.194 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 39 v 8 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-26T01:08:11.194 INFO:tasks.rados.rados.0.smithi164.stdout:150: finishing write tid 2 to smithi16428812-33 2024-04-26T01:08:11.194 INFO:tasks.rados.rados.0.smithi164.stdout:161: finishing write tid 1 to smithi16428812-7 2024-04-26T01:08:11.194 INFO:tasks.rados.rados.0.smithi164.stdout:150: finishing write tid 3 to smithi16428812-33 2024-04-26T01:08:11.194 INFO:tasks.rados.rados.0.smithi164.stdout:161: finishing write tid 2 to smithi16428812-7 2024-04-26T01:08:11.194 INFO:tasks.rados.rados.0.smithi164.stdout:161: finishing write tid 3 to smithi16428812-7 2024-04-26T01:08:11.194 INFO:tasks.rados.rados.0.smithi164.stdout:148: finishing copy_from to smithi16428812-47 2024-04-26T01:08:11.194 INFO:tasks.rados.rados.0.smithi164.stdout:148: got expected ENOENT (src dne) 2024-04-26T01:08:11.194 INFO:tasks.rados.rados.0.smithi164.stdout:171: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-26T01:08:11.200 INFO:tasks.rados.rados.0.smithi164.stdout:148: finishing copy_from racing read to smithi16428812-47 2024-04-26T01:08:11.200 INFO:tasks.rados.rados.0.smithi164.stdout:161: finishing write tid 4 to smithi16428812-7 2024-04-26T01:08:11.200 INFO:tasks.rados.rados.0.smithi164.stdout:161: oid 7 updating version 0 to 17 2024-04-26T01:08:11.200 INFO:tasks.rados.rados.0.smithi164.stdout:161: oid 7 updating version 17 to 18 2024-04-26T01:08:11.200 INFO:tasks.rados.rados.0.smithi164.stdout:161: oid 7 updating version 18 to 19 2024-04-26T01:08:11.200 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 7 v 19 (ObjNum 85 snap 6 seq_num 85) dirty exists 2024-04-26T01:08:11.200 INFO:tasks.rados.rados.0.smithi164.stdout:161: left oid 7 (ObjNum 85 snap 6 seq_num 85) 2024-04-26T01:08:11.200 INFO:tasks.rados.rados.0.smithi164.stdout:174: finishing write tid 1 to smithi16428812-42 2024-04-26T01:08:11.200 INFO:tasks.rados.rados.0.smithi164.stdout:174: finishing write tid 2 to smithi16428812-42 2024-04-26T01:08:11.200 INFO:tasks.rados.rados.0.smithi164.stdout:174: finishing write tid 3 to smithi16428812-42 2024-04-26T01:08:11.200 INFO:tasks.rados.rados.0.smithi164.stdout:174: finishing write tid 4 to smithi16428812-42 2024-04-26T01:08:11.200 INFO:tasks.rados.rados.0.smithi164.stdout:175: finishing copy_from racing read to smithi16428812-46 2024-04-26T01:08:11.200 INFO:tasks.rados.rados.0.smithi164.stdout:175: finishing copy_from to smithi16428812-46 2024-04-26T01:08:11.201 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 46 v 10 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-26T01:08:11.201 INFO:tasks.rados.rados.0.smithi164.stdout:176: finishing rollback tid 0 to smithi16428812-27 2024-04-26T01:08:11.201 INFO:tasks.rados.rados.0.smithi164.stdout:176: finishing rollback tid 1 to smithi16428812-27 2024-04-26T01:08:11.201 INFO:tasks.rados.rados.0.smithi164.stdout:142: done (14 left) 2024-04-26T01:08:11.201 INFO:tasks.rados.rados.0.smithi164.stdout:145: done (13 left) 2024-04-26T01:08:11.201 INFO:tasks.rados.rados.0.smithi164.stdout:148: done (12 left) 2024-04-26T01:08:11.201 INFO:tasks.rados.rados.0.smithi164.stdout:161: done (11 left) 2024-04-26T01:08:11.201 INFO:tasks.rados.rados.0.smithi164.stdout:164: done (10 left) 2024-04-26T01:08:11.201 INFO:tasks.rados.rados.0.smithi164.stdout:171: done (9 left) 2024-04-26T01:08:11.201 INFO:tasks.rados.rados.0.smithi164.stdout:173: done (8 left) 2024-04-26T01:08:11.201 INFO:tasks.rados.rados.0.smithi164.stdout:175: done (7 left) 2024-04-26T01:08:11.201 INFO:tasks.rados.rados.0.smithi164.stdout:177: done (6 left) 2024-04-26T01:08:11.201 INFO:tasks.rados.rados.0.smithi164.stdout:178: snap_create 2024-04-26T01:08:11.213 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:11.212+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:11.222 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:11.222 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:11.236 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:08:11.238 INFO:tasks.rados.rados.0.smithi164.stdout:153: finishing copy_from to smithi16428812-36 2024-04-26T01:08:11.239 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 36 v 10 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-26T01:08:11.315 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:11.312+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:11.325 INFO:tasks.rados.rados.0.smithi164.stdout:150: finishing write tid 4 to smithi16428812-33 2024-04-26T01:08:11.329 INFO:tasks.rados.rados.0.smithi164.stdout:150: finishing write tid 5 to smithi16428812-33 2024-04-26T01:08:11.330 INFO:tasks.rados.rados.0.smithi164.stdout:150: oid 33 updating version 0 to 30 2024-04-26T01:08:11.330 INFO:tasks.rados.rados.0.smithi164.stdout:150: oid 33 version 30 is already newer than 29 2024-04-26T01:08:11.330 INFO:tasks.rados.rados.0.smithi164.stdout:150: oid 33 version 30 is already newer than 28 2024-04-26T01:08:11.330 INFO:tasks.rados.rados.0.smithi164.stdout:150: oid 33 updating version 30 to 31 2024-04-26T01:08:11.330 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 33 v 31 (ObjNum 81 snap 5 seq_num 81) dirty exists 2024-04-26T01:08:11.330 INFO:tasks.rados.rados.0.smithi164.stdout:150: left oid 33 (ObjNum 81 snap 5 seq_num 81) 2024-04-26T01:08:11.371 INFO:tasks.rados.rados.0.smithi164.stdout:174: finishing write tid 5 to smithi16428812-42 2024-04-26T01:08:11.371 INFO:tasks.rados.rados.0.smithi164.stdout:174: oid 42 updating version 0 to 21 2024-04-26T01:08:11.371 INFO:tasks.rados.rados.0.smithi164.stdout:174: oid 42 updating version 21 to 22 2024-04-26T01:08:11.371 INFO:tasks.rados.rados.0.smithi164.stdout:174: oid 42 updating version 22 to 23 2024-04-26T01:08:11.371 INFO:tasks.rados.rados.0.smithi164.stdout:174: oid 42 version 23 is already newer than 20 2024-04-26T01:08:11.371 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 42 v 23 (ObjNum 91 snap 7 seq_num 91) dirty exists 2024-04-26T01:08:11.372 INFO:tasks.rados.rados.0.smithi164.stdout:174: left oid 42 (ObjNum 91 snap 7 seq_num 91) 2024-04-26T01:08:11.402 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:11.402 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:11.416 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:11.412+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:11.418 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:08:11.430 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-26T01:08:11.517 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:11.516+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:11.584 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:11.584 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:11.600 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:08:11.618 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:11.613+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:11.621 INFO:tasks.rados.rados.0.smithi164.stdout:176: finishing rollback tid 2 to smithi16428812-27 2024-04-26T01:08:11.621 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 27 v 22 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-26T01:08:11.657 INFO:teuthology.orchestra.run.smithi164.stdout:3.es3 2024-04-26T01:08:11.657 INFO:teuthology.orchestra.run.smithi164.stdout:3.cs0 2024-04-26T01:08:11.657 INFO:teuthology.orchestra.run.smithi164.stdout:3.as6 2024-04-26T01:08:11.658 INFO:teuthology.orchestra.run.smithi164.stdout:3.7s6 2024-04-26T01:08:11.658 INFO:teuthology.orchestra.run.smithi164.stdout:3.2s6 2024-04-26T01:08:11.658 INFO:teuthology.orchestra.run.smithi164.stdout:3.fs0 2024-04-26T01:08:11.658 INFO:teuthology.orchestra.run.smithi164.stdout:3.1s1 2024-04-26T01:08:11.717 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:11.717 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:11.721 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:11.716+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:11.732 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:08:11.825 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:11.820+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:11.868 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:11.868 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:11.883 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:08:11.898 INFO:tasks.rados.rados.0.smithi164.stdout:160: finishing write tid 4 to smithi16428812-18 2024-04-26T01:08:11.902 INFO:tasks.rados.rados.0.smithi164.stdout:160: finishing write tid 5 to smithi16428812-18 2024-04-26T01:08:11.902 INFO:tasks.rados.rados.0.smithi164.stdout:160: oid 18 updating version 0 to 28 2024-04-26T01:08:11.902 INFO:tasks.rados.rados.0.smithi164.stdout:160: oid 18 version 28 is already newer than 27 2024-04-26T01:08:11.902 INFO:tasks.rados.rados.0.smithi164.stdout:160: oid 18 version 28 is already newer than 26 2024-04-26T01:08:11.902 INFO:tasks.rados.rados.0.smithi164.stdout:160: oid 18 updating version 28 to 30 2024-04-26T01:08:11.902 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 18 v 30 (ObjNum 84 snap 6 seq_num 84) dirty exists 2024-04-26T01:08:11.903 INFO:tasks.rados.rados.0.smithi164.stdout:160: left oid 18 (ObjNum 84 snap 6 seq_num 84) 2024-04-26T01:08:11.928 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:11.925+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:11.931 INFO:tasks.rados.rados.0.smithi164.stdout:172: finishing copy_from to smithi16428812-17 2024-04-26T01:08:11.931 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 17 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-26T01:08:12.006 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:12.006 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:12.022 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:08:12.024 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:12.024+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:12.032 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-04-26T01:08:12.036 INFO:tasks.rados.rados.0.smithi164.stdout:150: done (6 left) 2024-04-26T01:08:12.036 INFO:tasks.rados.rados.0.smithi164.stdout:153: done (5 left) 2024-04-26T01:08:12.036 INFO:tasks.rados.rados.0.smithi164.stdout:160: done (4 left) 2024-04-26T01:08:12.036 INFO:tasks.rados.rados.0.smithi164.stdout:172: done (3 left) 2024-04-26T01:08:12.036 INFO:tasks.rados.rados.0.smithi164.stdout:174: done (2 left) 2024-04-26T01:08:12.036 INFO:tasks.rados.rados.0.smithi164.stdout:176: done (1 left) 2024-04-26T01:08:12.037 INFO:tasks.rados.rados.0.smithi164.stdout:178: done (0 left) 2024-04-26T01:08:12.037 INFO:tasks.rados.rados.0.smithi164.stdout:179: delete oid 9 current snap is 8 2024-04-26T01:08:12.049 INFO:tasks.rados.rados.0.smithi164.stdout:179: done (0 left) 2024-04-26T01:08:12.049 INFO:tasks.rados.rados.0.smithi164.stdout:180: delete oid 25 current snap is 8 2024-04-26T01:08:12.060 INFO:tasks.rados.rados.0.smithi164.stdout:180: done (0 left) 2024-04-26T01:08:12.060 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 22 current snap is 8 2024-04-26T01:08:12.060 INFO:tasks.rados.rados.0.smithi164.stdout:181: seq_num 95 ranges {360448=442368,802816=540672,1343488=622592} 2024-04-26T01:08:12.073 INFO:tasks.rados.rados.0.smithi164.stdout:181: writing smithi16428812-22 from 360448 to 802816 tid 1 2024-04-26T01:08:12.082 INFO:tasks.rados.rados.0.smithi164.stdout:181: writing smithi16428812-22 from 802816 to 1343488 tid 2 2024-04-26T01:08:12.092 INFO:tasks.rados.rados.0.smithi164.stdout:181: writing smithi16428812-22 from 1343488 to 1966080 tid 3 2024-04-26T01:08:12.092 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 3 current snap is 8 2024-04-26T01:08:12.092 INFO:tasks.rados.rados.0.smithi164.stdout:182: seq_num 96 ranges {0=458752,458752=688128,1146880=475136} 2024-04-26T01:08:12.099 INFO:tasks.rados.rados.0.smithi164.stdout:182: writing smithi16428812-3 from 0 to 458752 tid 1 2024-04-26T01:08:12.110 INFO:tasks.rados.rados.0.smithi164.stdout:182: writing smithi16428812-3 from 458752 to 1146880 tid 2 2024-04-26T01:08:12.116 INFO:tasks.rados.rados.0.smithi164.stdout:182: writing smithi16428812-3 from 1146880 to 1622016 tid 3 2024-04-26T01:08:12.117 INFO:tasks.rados.rados.0.smithi164.stdout:183: snap_remove snap 7 2024-04-26T01:08:12.124 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:12.124+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:12.193 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-11 --journal-path /var/lib/ceph/osd/ceph-11/journal --op export --pgid 3.as6 --file /var/log/ceph/exp.3.as6.11 2024-04-26T01:08:12.195 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:12.195 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:12.211 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:08:12.225 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:12.220+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:12.326 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:12.324+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:12.338 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:12.338 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:12.352 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:08:12.427 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:12.424+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:12.479 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:12.479 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:12.493 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:08:12.540 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:12.537+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:12.610 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:12.610 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:12.624 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:08:12.629 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:12.624+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:12.729 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:12.728+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:12.796 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:12.797 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:12.811 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:08:12.830 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:12.828+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:12.930 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:12.930 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:12.931 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:12.928+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:12.944 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:08:13.032 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:13.029+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:13.042 INFO:tasks.rados.rados.0.smithi164.stdout:183: done (2 left) 2024-04-26T01:08:13.042 INFO:tasks.rados.rados.0.smithi164.stdout:184: rollback oid 49 current snap is 8 2024-04-26T01:08:13.042 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 49 to 6 2024-04-26T01:08:13.042 INFO:tasks.rados.rados.0.smithi164.stdout:181: finishing write tid 1 to smithi16428812-22 2024-04-26T01:08:13.042 INFO:tasks.rados.rados.0.smithi164.stdout:181: finishing write tid 2 to smithi16428812-22 2024-04-26T01:08:13.042 INFO:tasks.rados.rados.0.smithi164.stdout:181: finishing write tid 3 to smithi16428812-22 2024-04-26T01:08:13.042 INFO:tasks.rados.rados.0.smithi164.stdout:181: finishing write tid 4 to smithi16428812-22 2024-04-26T01:08:13.042 INFO:tasks.rados.rados.0.smithi164.stdout:181: finishing write tid 5 to smithi16428812-22 2024-04-26T01:08:13.042 INFO:tasks.rados.rados.0.smithi164.stdout:181: oid 22 updating version 0 to 28 2024-04-26T01:08:13.043 INFO:tasks.rados.rados.0.smithi164.stdout:181: oid 22 version 28 is already newer than 25 2024-04-26T01:08:13.043 INFO:tasks.rados.rados.0.smithi164.stdout:181: oid 22 version 28 is already newer than 26 2024-04-26T01:08:13.043 INFO:tasks.rados.rados.0.smithi164.stdout:181: oid 22 version 28 is already newer than 27 2024-04-26T01:08:13.043 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 22 v 28 (ObjNum 95 snap 8 seq_num 95) dirty exists 2024-04-26T01:08:13.043 INFO:tasks.rados.rados.0.smithi164.stdout:181: left oid 22 (ObjNum 95 snap 8 seq_num 95) 2024-04-26T01:08:13.043 INFO:tasks.rados.rados.0.smithi164.stdout:182: finishing write tid 1 to smithi16428812-3 2024-04-26T01:08:13.043 INFO:tasks.rados.rados.0.smithi164.stdout:182: finishing write tid 2 to smithi16428812-3 2024-04-26T01:08:13.043 INFO:tasks.rados.rados.0.smithi164.stdout:182: finishing write tid 3 to smithi16428812-3 2024-04-26T01:08:13.043 INFO:tasks.rados.rados.0.smithi164.stdout:182: finishing write tid 4 to smithi16428812-3 2024-04-26T01:08:13.043 INFO:tasks.rados.rados.0.smithi164.stdout:182: finishing write tid 5 to smithi16428812-3 2024-04-26T01:08:13.043 INFO:tasks.rados.rados.0.smithi164.stdout:182: oid 3 updating version 0 to 24 2024-04-26T01:08:13.043 INFO:tasks.rados.rados.0.smithi164.stdout:182: oid 3 version 24 is already newer than 23 2024-04-26T01:08:13.043 INFO:tasks.rados.rados.0.smithi164.stdout:182: oid 3 version 24 is already newer than 21 2024-04-26T01:08:13.043 INFO:tasks.rados.rados.0.smithi164.stdout:182: oid 3 version 24 is already newer than 22 2024-04-26T01:08:13.044 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 3 v 24 (ObjNum 96 snap 8 seq_num 96) dirty exists 2024-04-26T01:08:13.044 INFO:tasks.rados.rados.0.smithi164.stdout:182: left oid 3 (ObjNum 96 snap 8 seq_num 96) 2024-04-26T01:08:13.044 INFO:tasks.rados.rados.0.smithi164.stdout:181: done (2 left) 2024-04-26T01:08:13.044 INFO:tasks.rados.rados.0.smithi164.stdout:182: done (1 left) 2024-04-26T01:08:13.044 INFO:tasks.rados.rados.0.smithi164.stdout:185: rollback oid 29 current snap is 8 2024-04-26T01:08:13.044 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 29 to 5 2024-04-26T01:08:13.044 INFO:tasks.rados.rados.0.smithi164.stdout:186: read oid 3 snap -1 2024-04-26T01:08:13.044 INFO:tasks.rados.rados.0.smithi164.stdout:186: expect (ObjNum 96 snap 8 seq_num 96) 2024-04-26T01:08:13.044 INFO:tasks.rados.rados.0.smithi164.stdout:187: snap_create 2024-04-26T01:08:13.058 INFO:tasks.rados.rados.0.smithi164.stdout:186: expect (ObjNum 96 snap 8 seq_num 96) 2024-04-26T01:08:13.074 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:13.074 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:13.091 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:08:13.104 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:08:13.124 INFO:tasks.rados.rados.0.smithi164.stdout:185: finishing rollback tid 0 to smithi16428812-29 2024-04-26T01:08:13.124 INFO:tasks.rados.rados.0.smithi164.stdout:184: finishing rollback tid 0 to smithi16428812-49 2024-04-26T01:08:13.124 INFO:tasks.rados.rados.0.smithi164.stdout:185: finishing rollback tid 1 to smithi16428812-29 2024-04-26T01:08:13.124 INFO:tasks.rados.rados.0.smithi164.stdout:185: finishing rollback tid 2 to smithi16428812-29 2024-04-26T01:08:13.124 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 29 v 16 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-26T01:08:13.124 INFO:tasks.rados.rados.0.smithi164.stdout:184: finishing rollback tid 1 to smithi16428812-49 2024-04-26T01:08:13.124 INFO:tasks.rados.rados.0.smithi164.stdout:184: finishing rollback tid 2 to smithi16428812-49 2024-04-26T01:08:13.124 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 49 v 34 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-26T01:08:13.132 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:13.132+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:13.233 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:13.229+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:13.250 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:13.250 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:13.264 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:08:13.272 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:13.272 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:13.297 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:13.297 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:13.326 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:13.326 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:13.352 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:13.349+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:13.355 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:13.355 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:Exporting 3.as6 info 3.as6( v 41'10 (0'0,41'10] local-lis/les=30/31 n=1 ec=30/30 lis/c=30/30 les/c/f=31/31/0 sis=30) 2024-04-26T01:08:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:Read 6#3:592ac288:::smithi16428812-20:head# 2024-04-26T01:08:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:Export successful 2024-04-26T01:08:13.383 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:13.383 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:13.414 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:13.414 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:13.435 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:13.432+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:13.437 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:13.437 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:13.452 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:08:13.536 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:13.536+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:13.546 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:13.546 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:13.584 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:13.585 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:13.624 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:13.624 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:13.631 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:13.631 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:13.637 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:13.633+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:13.638 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:08:13.674 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:13.674 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:13.714 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:13.714 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:13.715 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-26T01:08:13.737 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:13.732+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:13.756 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:13.756 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:13.769 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:13.770 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:13.770 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:08:13.770 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:13.783 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:08:13.796 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:13.796 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:13.834 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:13.834 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:13.837 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:13.836+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:13.874 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:13.872+0000 7f5813fff640 0 -- 172.21.15.8:0/3982344105 >> [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] conn(0x7f581c223620 msgr2=0x7f581c225a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:08:13.883 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:13.889 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T01:08:13.889 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T01:08:13.901 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:13.901 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:13.901 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:08:13.901 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:08:13.901 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:13.914 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:08:13.938 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:13.940+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:13.947 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-11 --journal-path /var/lib/ceph/osd/ceph-11/journal --force --op remove --pgid 3.as6 2024-04-26T01:08:14.039 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:14.040+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:14.042 INFO:tasks.rados.rados.0.smithi164.stdout:184: done (3 left) 2024-04-26T01:08:14.042 INFO:tasks.rados.rados.0.smithi164.stdout:185: done (2 left) 2024-04-26T01:08:14.042 INFO:tasks.rados.rados.0.smithi164.stdout:186: done (1 left) 2024-04-26T01:08:14.042 INFO:tasks.rados.rados.0.smithi164.stdout:187: done (0 left) 2024-04-26T01:08:14.043 INFO:tasks.rados.rados.0.smithi164.stdout:188: copy_from oid 29 from oid 50 current snap is 9 2024-04-26T01:08:14.043 INFO:tasks.rados.rados.0.smithi164.stdout:189: snap_create 2024-04-26T01:08:14.058 INFO:tasks.rados.rados.0.smithi164.stdout:188: finishing copy_from to smithi16428812-29 2024-04-26T01:08:14.058 INFO:tasks.rados.rados.0.smithi164.stdout:188: got expected ENOENT (src dne) 2024-04-26T01:08:14.059 INFO:tasks.rados.rados.0.smithi164.stdout:188: finishing copy_from racing read to smithi16428812-29 2024-04-26T01:08:14.060 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-26T01:08:14.082 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:14.083 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:08:14.083 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:08:14.083 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:08:14.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.083 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:145 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'103, mlcod=25'98)", 2024-04-26T01:08:14.083 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:47.983085+0000", 2024-04-26T01:08:14.083 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 26.097124206, 2024-04-26T01:08:14.083 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.055794923000000003, 2024-04-26T01:08:14.083 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.083 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.083 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:14.083 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.084 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.084 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983085+0000", 2024-04-26T01:08:14.084 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.084 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.084 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983085+0000", 2024-04-26T01:08:14.084 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.084 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.084 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983086+0000", 2024-04-26T01:08:14.084 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9699999999999994e-07 2024-04-26T01:08:14.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.085 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.085 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983194+0000", 2024-04-26T01:08:14.085 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107569 2024-04-26T01:08:14.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.085 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.085 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037255+0000", 2024-04-26T01:08:14.085 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.054061302999999998 2024-04-26T01:08:14.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.085 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.085 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037268+0000", 2024-04-26T01:08:14.085 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3477e-05 2024-04-26T01:08:14.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.086 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.086 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037378+0000", 2024-04-26T01:08:14.086 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000110089 2024-04-26T01:08:14.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.086 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:14.086 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037414+0000", 2024-04-26T01:08:14.086 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5123000000000002e-05 2024-04-26T01:08:14.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.086 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:14.086 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.038790+0000", 2024-04-26T01:08:14.086 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0013768820000000001 2024-04-26T01:08:14.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.087 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.087 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.038880+0000", 2024-04-26T01:08:14.087 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9482999999999997e-05 2024-04-26T01:08:14.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.087 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.087 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:22 3.es0 3:7b1f3e32:::smithi16428812-8:head [append 0~131072,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:14.087 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.308640+0000", 2024-04-26T01:08:14.087 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.771569438, 2024-04-26T01:08:14.087 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014721699, 2024-04-26T01:08:14.088 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.088 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.088 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:14.088 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:14.088 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:14.088 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:14.088 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 22 2024-04-26T01:08:14.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.088 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.088 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.088 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.308640+0000", 2024-04-26T01:08:14.088 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.089 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.089 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.089 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.308640+0000", 2024-04-26T01:08:14.089 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.089 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.089 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.089 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.308642+0000", 2024-04-26T01:08:14.089 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.968e-06 2024-04-26T01:08:14.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.089 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.089 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.089 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309326+0000", 2024-04-26T01:08:14.090 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00068456400000000003 2024-04-26T01:08:14.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.090 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.090 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.090 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309343+0000", 2024-04-26T01:08:14.090 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6262000000000001e-05 2024-04-26T01:08:14.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.090 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.090 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.090 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309353+0000", 2024-04-26T01:08:14.090 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0414999999999999e-05 2024-04-26T01:08:14.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.090 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.091 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.091 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309627+0000", 2024-04-26T01:08:14.091 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000273952 2024-04-26T01:08:14.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.091 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:14.091 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309802+0000", 2024-04-26T01:08:14.091 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017508700000000001 2024-04-26T01:08:14.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.091 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.091 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.310721+0000", 2024-04-26T01:08:14.091 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00091951000000000001 2024-04-26T01:08:14.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.092 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.092 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314042+0000", 2024-04-26T01:08:14.092 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0033201860000000001 2024-04-26T01:08:14.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.092 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:14.092 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.323251+0000", 2024-04-26T01:08:14.092 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0092089760000000007 2024-04-26T01:08:14.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.092 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.092 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.323361+0000", 2024-04-26T01:08:14.092 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000110779 2024-04-26T01:08:14.093 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.093 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.093 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.093 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.093 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:23 3.es0 3:7b1f3e32:::smithi16428812-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:14.093 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.309394+0000", 2024-04-26T01:08:14.093 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.770815388000001, 2024-04-26T01:08:14.093 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014254325999999999, 2024-04-26T01:08:14.093 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.093 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.093 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:14.093 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:14.093 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:14.093 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:14.094 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 23 2024-04-26T01:08:14.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.094 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.094 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.094 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.094 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309394+0000", 2024-04-26T01:08:14.094 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.094 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.094 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.094 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309394+0000", 2024-04-26T01:08:14.094 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.094 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.095 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.095 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309395+0000", 2024-04-26T01:08:14.095 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6199999999999999e-06 2024-04-26T01:08:14.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.095 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.095 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.095 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309401+0000", 2024-04-26T01:08:14.095 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3070000000000002e-06 2024-04-26T01:08:14.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.095 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.095 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.095 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309412+0000", 2024-04-26T01:08:14.095 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1635e-05 2024-04-26T01:08:14.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.096 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.096 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.096 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309421+0000", 2024-04-26T01:08:14.096 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.0790000000000007e-06 2024-04-26T01:08:14.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.096 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.096 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.096 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.311393+0000", 2024-04-26T01:08:14.096 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0019715599999999998 2024-04-26T01:08:14.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.096 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.096 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:14.096 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.311556+0000", 2024-04-26T01:08:14.096 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016265600000000001 2024-04-26T01:08:14.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.097 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.097 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.097 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312482+0000", 2024-04-26T01:08:14.097 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00092630499999999999 2024-04-26T01:08:14.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.097 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.097 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.097 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314656+0000", 2024-04-26T01:08:14.097 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0021742110000000001 2024-04-26T01:08:14.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.097 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.097 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:14.097 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.323528+0000", 2024-04-26T01:08:14.098 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0088717139999999993 2024-04-26T01:08:14.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.098 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.098 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.098 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.323648+0000", 2024-04-26T01:08:14.098 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000120239 2024-04-26T01:08:14.098 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.098 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.098 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.098 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.098 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:24 3.es0 3:7b1f3e32:::smithi16428812-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:14.098 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.309434+0000", 2024-04-26T01:08:14.099 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.770775177000001, 2024-04-26T01:08:14.099 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.019012134999999999, 2024-04-26T01:08:14.099 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.099 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.099 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:14.099 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:14.099 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:14.099 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:14.099 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 24 2024-04-26T01:08:14.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.099 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.099 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.099 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309434+0000", 2024-04-26T01:08:14.100 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.100 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.100 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309434+0000", 2024-04-26T01:08:14.100 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.100 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.100 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309435+0000", 2024-04-26T01:08:14.100 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.3e-07 2024-04-26T01:08:14.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.100 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.101 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309439+0000", 2024-04-26T01:08:14.101 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8530000000000002e-06 2024-04-26T01:08:14.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.101 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.101 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309448+0000", 2024-04-26T01:08:14.101 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9879999999999993e-06 2024-04-26T01:08:14.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.101 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.101 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309454+0000", 2024-04-26T01:08:14.101 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.6630000000000004e-06 2024-04-26T01:08:14.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.102 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.102 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312756+0000", 2024-04-26T01:08:14.102 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003301712 2024-04-26T01:08:14.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.102 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:14.102 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312870+0000", 2024-04-26T01:08:14.102 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000113983 2024-04-26T01:08:14.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.102 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.102 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.324091+0000", 2024-04-26T01:08:14.102 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011221419 2024-04-26T01:08:14.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.103 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:14.103 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.324213+0000", 2024-04-26T01:08:14.103 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012186200000000001 2024-04-26T01:08:14.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.103 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.103 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.328446+0000", 2024-04-26T01:08:14.103 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0042329250000000002 2024-04-26T01:08:14.103 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.103 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.103 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.104 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpRead(3.2s1 33/30 ECSubRead(tid=163, to_read={3:410d186f:::smithi16428812-11:head=0,4096,0}, subchunks={3:410d186f:::smithi16428812-11:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:14.104 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.340790+0000", 2024-04-26T01:08:14.104 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.739419208999999, 2024-04-26T01:08:14.104 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.047962901000000002, 2024-04-26T01:08:14.104 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.104 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.104 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.104 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.104 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.104 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340790+0000", 2024-04-26T01:08:14.104 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.105 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.105 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340790+0000", 2024-04-26T01:08:14.105 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.105 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.105 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.105 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340791+0000", 2024-04-26T01:08:14.105 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.3300000000000001e-07 2024-04-26T01:08:14.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.105 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.105 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.105 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340801+0000", 2024-04-26T01:08:14.105 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0308e-05 2024-04-26T01:08:14.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.106 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.106 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388316+0000", 2024-04-26T01:08:14.106 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.047514702999999998 2024-04-26T01:08:14.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.106 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.106 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388324+0000", 2024-04-26T01:08:14.106 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.3529999999999995e-06 2024-04-26T01:08:14.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.106 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.106 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388439+0000", 2024-04-26T01:08:14.106 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000114657 2024-04-26T01:08:14.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.107 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.107 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.107 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388753+0000", 2024-04-26T01:08:14.107 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00031404700000000002 2024-04-26T01:08:14.107 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.107 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.107 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.107 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.107 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:62 3.es0 3:7eef64f7:::smithi16428812-21:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:14.107 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.361875+0000", 2024-04-26T01:08:14.107 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.718334134999999, 2024-04-26T01:08:14.107 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.111184554, 2024-04-26T01:08:14.108 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.108 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.108 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:14.108 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:14.108 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:14.108 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:14.108 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 62 2024-04-26T01:08:14.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.108 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.108 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.108 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.108 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361875+0000", 2024-04-26T01:08:14.108 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.109 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.109 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361875+0000", 2024-04-26T01:08:14.109 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.109 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.109 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361876+0000", 2024-04-26T01:08:14.109 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5899999999999995e-07 2024-04-26T01:08:14.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.109 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.109 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361880+0000", 2024-04-26T01:08:14.109 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8739999999999997e-06 2024-04-26T01:08:14.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.110 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.110 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361887+0000", 2024-04-26T01:08:14.110 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4839999999999998e-06 2024-04-26T01:08:14.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.110 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.110 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361892+0000", 2024-04-26T01:08:14.110 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.437e-06 2024-04-26T01:08:14.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.110 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.110 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.363374+0000", 2024-04-26T01:08:14.111 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001482693 2024-04-26T01:08:14.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.111 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:14.111 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.363442+0000", 2024-04-26T01:08:14.111 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7448999999999997e-05 2024-04-26T01:08:14.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.111 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.111 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.363805+0000", 2024-04-26T01:08:14.111 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00036289900000000002 2024-04-26T01:08:14.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.111 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.111 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.366199+0000", 2024-04-26T01:08:14.112 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.002394654 2024-04-26T01:08:14.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.112 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.112 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:14.112 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472968+0000", 2024-04-26T01:08:14.112 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.106769087 2024-04-26T01:08:14.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.112 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.112 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.112 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473060+0000", 2024-04-26T01:08:14.112 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.1218000000000007e-05 2024-04-26T01:08:14.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.112 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.113 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.113 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:63 3.es0 3:7eef64f7:::smithi16428812-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:14.113 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.361913+0000", 2024-04-26T01:08:14.113 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.718296191, 2024-04-26T01:08:14.113 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.11409399200000001, 2024-04-26T01:08:14.113 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.113 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.113 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:14.113 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:14.113 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:14.113 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:14.113 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 63 2024-04-26T01:08:14.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.114 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.114 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.114 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.114 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361913+0000", 2024-04-26T01:08:14.114 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.114 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.114 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.114 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361913+0000", 2024-04-26T01:08:14.114 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.114 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.114 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.115 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361914+0000", 2024-04-26T01:08:14.115 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.5700000000000002e-07 2024-04-26T01:08:14.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.115 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.115 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.115 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361917+0000", 2024-04-26T01:08:14.115 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0979999999999998e-06 2024-04-26T01:08:14.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.115 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.115 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.115 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361923+0000", 2024-04-26T01:08:14.115 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9719999999999997e-06 2024-04-26T01:08:14.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.116 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.116 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.116 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361927+0000", 2024-04-26T01:08:14.116 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.5070000000000002e-06 2024-04-26T01:08:14.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.116 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.116 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.116 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.364059+0000", 2024-04-26T01:08:14.116 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.002132278 2024-04-26T01:08:14.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.116 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.116 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:14.116 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.364206+0000", 2024-04-26T01:08:14.116 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014703700000000001 2024-04-26T01:08:14.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.117 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.117 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473123+0000", 2024-04-26T01:08:14.117 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.108916164 2024-04-26T01:08:14.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.117 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:14.117 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473211+0000", 2024-04-26T01:08:14.117 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8381999999999995e-05 2024-04-26T01:08:14.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.117 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.117 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.476007+0000", 2024-04-26T01:08:14.118 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0027959970000000002 2024-04-26T01:08:14.118 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.118 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.118 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.118 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.118 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWriteReply(3.es0 33/30 ECSubWriteReply(tid=5, last_complete=33'4, committed=1, applied=1))", 2024-04-26T01:08:14.118 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.365083+0000", 2024-04-26T01:08:14.118 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.715125709, 2024-04-26T01:08:14.118 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.107988984, 2024-04-26T01:08:14.118 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.118 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.118 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.118 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.119 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.119 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365083+0000", 2024-04-26T01:08:14.119 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.119 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.119 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365083+0000", 2024-04-26T01:08:14.119 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.119 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.119 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365084+0000", 2024-04-26T01:08:14.119 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.9299999999999999e-07 2024-04-26T01:08:14.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.120 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.120 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.120 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365086+0000", 2024-04-26T01:08:14.120 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5239999999999999e-06 2024-04-26T01:08:14.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.120 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.120 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.120 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472672+0000", 2024-04-26T01:08:14.120 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.10758553899999999 2024-04-26T01:08:14.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.120 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.120 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.120 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472680+0000", 2024-04-26T01:08:14.121 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.6820000000000003e-06 2024-04-26T01:08:14.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.121 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.121 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.121 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472900+0000", 2024-04-26T01:08:14.121 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00022041899999999999 2024-04-26T01:08:14.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.121 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.121 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.121 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473072+0000", 2024-04-26T01:08:14.121 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017242700000000001 2024-04-26T01:08:14.121 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.121 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.121 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.122 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.2s1 33/30 ECSubWrite(tid=165, reqid=client.4521.0:80, at_version=33'6, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:08:14.122 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.391943+0000", 2024-04-26T01:08:14.122 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.688266031, 2024-04-26T01:08:14.122 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.042848432999999998, 2024-04-26T01:08:14.122 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.122 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.122 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.122 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.122 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.122 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391943+0000", 2024-04-26T01:08:14.122 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.123 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.123 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391943+0000", 2024-04-26T01:08:14.123 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.123 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.123 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391944+0000", 2024-04-26T01:08:14.123 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8199999999999998e-07 2024-04-26T01:08:14.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.123 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.123 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391961+0000", 2024-04-26T01:08:14.124 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6546999999999999e-05 2024-04-26T01:08:14.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.124 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.124 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.124 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.430981+0000", 2024-04-26T01:08:14.124 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.039020817999999999 2024-04-26T01:08:14.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.124 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.124 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.124 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.430991+0000", 2024-04-26T01:08:14.124 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7059999999999999e-06 2024-04-26T01:08:14.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.124 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.125 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.125 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.431059+0000", 2024-04-26T01:08:14.125 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.8198e-05 2024-04-26T01:08:14.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.125 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.125 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.125 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.431067+0000", 2024-04-26T01:08:14.125 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.8930000000000005e-06 2024-04-26T01:08:14.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.125 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.125 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.125 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.434752+0000", 2024-04-26T01:08:14.125 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0036849249999999999 2024-04-26T01:08:14.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.125 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.126 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.126 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.434792+0000", 2024-04-26T01:08:14.126 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.9464e-05 2024-04-26T01:08:14.126 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.126 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.126 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.126 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.126 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.3s4 33/30 ECSubWrite(tid=170, reqid=client.4521.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=33'2))", 2024-04-26T01:08:14.126 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.441271+0000", 2024-04-26T01:08:14.126 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.638938252999999, 2024-04-26T01:08:14.126 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.063989213000000003, 2024-04-26T01:08:14.126 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.127 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.127 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.127 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.127 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.127 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.127 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441271+0000", 2024-04-26T01:08:14.127 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.127 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.127 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.127 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441271+0000", 2024-04-26T01:08:14.127 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.127 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441272+0000", 2024-04-26T01:08:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1650000000000001e-06 2024-04-26T01:08:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441286+0000", 2024-04-26T01:08:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3771000000000001e-05 2024-04-26T01:08:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504639+0000", 2024-04-26T01:08:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.063352787999999993 2024-04-26T01:08:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.129 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.129 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504647+0000", 2024-04-26T01:08:14.129 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9389999999999997e-06 2024-04-26T01:08:14.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.129 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.129 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.129 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504715+0000", 2024-04-26T01:08:14.129 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7926e-05 2024-04-26T01:08:14.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.129 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.129 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.129 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504722+0000", 2024-04-26T01:08:14.129 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9449999999999993e-06 2024-04-26T01:08:14.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.130 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.130 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.130 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.505203+0000", 2024-04-26T01:08:14.130 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00048092399999999998 2024-04-26T01:08:14.130 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.130 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.130 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.130 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.505260+0000", 2024-04-26T01:08:14.130 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.6755000000000001e-05 2024-04-26T01:08:14.130 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.130 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.130 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.130 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.130 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.131 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:233 3.es0 3:7b1f3e32:::smithi16428812-8:head [copy-from ver 4] snapc 3=[2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-26T01:08:14.131 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:02.464122+0000", 2024-04-26T01:08:14.131 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.616087610999999, 2024-04-26T01:08:14.131 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.66639822699999995, 2024-04-26T01:08:14.131 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.131 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.131 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:14.131 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:14.131 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:14.131 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:14.131 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 233 2024-04-26T01:08:14.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.131 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.132 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.132 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.132 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464122+0000", 2024-04-26T01:08:14.132 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.132 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.132 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.132 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.132 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464122+0000", 2024-04-26T01:08:14.132 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.132 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.132 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.132 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.132 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464125+0000", 2024-04-26T01:08:14.132 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0939999999999999e-06 2024-04-26T01:08:14.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.133 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.133 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.133 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464136+0000", 2024-04-26T01:08:14.133 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1790000000000001e-05 2024-04-26T01:08:14.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.133 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.133 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.133 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464160+0000", 2024-04-26T01:08:14.133 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3683e-05 2024-04-26T01:08:14.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.133 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.133 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.133 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464174+0000", 2024-04-26T01:08:14.134 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.382e-05 2024-04-26T01:08:14.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.134 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.134 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.134 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464324+0000", 2024-04-26T01:08:14.134 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014991900000000001 2024-04-26T01:08:14.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.134 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.134 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:14.134 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464485+0000", 2024-04-26T01:08:14.134 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016099500000000001 2024-04-26T01:08:14.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.134 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.134 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.135 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480641+0000", 2024-04-26T01:08:14.135 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016155748000000001 2024-04-26T01:08:14.135 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.135 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.135 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.135 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.485807+0000", 2024-04-26T01:08:14.135 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0051663380000000004 2024-04-26T01:08:14.135 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.135 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.135 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:14.135 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.130287+0000", 2024-04-26T01:08:14.135 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.64448033500000002 2024-04-26T01:08:14.135 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.130520+0000", 2024-04-26T01:08:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00023250500000000001 2024-04-26T01:08:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:234 3.es0 3:7b1f3e32:::smithi16428812-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-04-26T01:08:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:02.464220+0000", 2024-04-26T01:08:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.615989268, 2024-04-26T01:08:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.019428925, 2024-04-26T01:08:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.137 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:14.137 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:14.137 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:14.137 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:14.137 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 234 2024-04-26T01:08:14.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.137 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.137 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.137 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464220+0000", 2024-04-26T01:08:14.137 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.137 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.138 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464220+0000", 2024-04-26T01:08:14.138 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.138 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.138 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.138 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464221+0000", 2024-04-26T01:08:14.138 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4500000000000001e-06 2024-04-26T01:08:14.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.138 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.138 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.138 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464229+0000", 2024-04-26T01:08:14.138 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.6229999999999997e-06 2024-04-26T01:08:14.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.139 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.139 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.139 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464250+0000", 2024-04-26T01:08:14.139 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1163e-05 2024-04-26T01:08:14.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.139 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.139 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.139 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464267+0000", 2024-04-26T01:08:14.139 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.668e-05 2024-04-26T01:08:14.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.139 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.139 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.139 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464815+0000", 2024-04-26T01:08:14.139 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00054803000000000002 2024-04-26T01:08:14.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.140 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.140 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for blocked object", 2024-04-26T01:08:14.140 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464939+0000", 2024-04-26T01:08:14.140 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000124385 2024-04-26T01:08:14.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.140 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.140 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.140 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.483264+0000", 2024-04-26T01:08:14.140 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.018324427000000001 2024-04-26T01:08:14.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.140 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.140 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:14.140 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.483429+0000", 2024-04-26T01:08:14.140 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000165749 2024-04-26T01:08:14.141 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.141 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.141 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.141 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.483649+0000", 2024-04-26T01:08:14.141 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00021941800000000001 2024-04-26T01:08:14.141 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.141 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.141 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.141 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.141 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.141 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:256 3.es0 3:7b1f3e32:::smithi16428812-8:head [rollback 2] snapc 4=[4,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:14.141 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:03.446889+0000", 2024-04-26T01:08:14.141 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.633320461, 2024-04-26T01:08:14.141 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.015497489, 2024-04-26T01:08:14.142 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.142 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.142 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:14.142 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:14.142 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:14.142 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:14.142 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 256 2024-04-26T01:08:14.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.142 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.142 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.142 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.142 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446889+0000", 2024-04-26T01:08:14.142 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.143 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.143 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.143 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446889+0000", 2024-04-26T01:08:14.143 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.143 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.143 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.143 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446889+0000", 2024-04-26T01:08:14.143 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.0999999999999998e-07 2024-04-26T01:08:14.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.143 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.143 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.143 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446893+0000", 2024-04-26T01:08:14.143 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7500000000000001e-06 2024-04-26T01:08:14.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.144 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.144 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.144 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446902+0000", 2024-04-26T01:08:14.144 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9230000000000008e-06 2024-04-26T01:08:14.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.144 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.144 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.144 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446909+0000", 2024-04-26T01:08:14.144 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.3429999999999998e-06 2024-04-26T01:08:14.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.144 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.144 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.144 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.456036+0000", 2024-04-26T01:08:14.145 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0091262870000000003 2024-04-26T01:08:14.145 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.145 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.145 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:14.145 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.456364+0000", 2024-04-26T01:08:14.145 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00032822100000000001 2024-04-26T01:08:14.145 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.145 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.145 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.145 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.457860+0000", 2024-04-26T01:08:14.145 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0014959909999999999 2024-04-26T01:08:14.145 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.145 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.145 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.146 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461062+0000", 2024-04-26T01:08:14.146 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0032019650000000002 2024-04-26T01:08:14.146 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.146 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.146 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:14.146 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.462190+0000", 2024-04-26T01:08:14.146 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001127954 2024-04-26T01:08:14.146 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.146 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.146 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.146 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.462386+0000", 2024-04-26T01:08:14.146 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000196345 2024-04-26T01:08:14.146 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.146 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.147 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.147 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.147 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:257 3.es0 3:7b1f3e32:::smithi16428812-8:head [append 0~0] snapc 4=[4,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:14.147 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:03.446919+0000", 2024-04-26T01:08:14.147 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.633290302000001, 2024-04-26T01:08:14.147 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.84360518900000003, 2024-04-26T01:08:14.147 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.147 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.147 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:14.147 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:14.147 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:14.147 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:14.147 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 257 2024-04-26T01:08:14.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.148 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.148 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.148 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.148 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446919+0000", 2024-04-26T01:08:14.148 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.148 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.148 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.148 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446919+0000", 2024-04-26T01:08:14.148 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.148 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.148 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.148 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446920+0000", 2024-04-26T01:08:14.148 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4700000000000001e-07 2024-04-26T01:08:14.149 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.149 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.149 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.149 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446922+0000", 2024-04-26T01:08:14.149 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6800000000000002e-06 2024-04-26T01:08:14.149 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.149 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.149 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.149 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446931+0000", 2024-04-26T01:08:14.149 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8659999999999995e-06 2024-04-26T01:08:14.149 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.149 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.149 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.149 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446941+0000", 2024-04-26T01:08:14.150 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.5540000000000003e-06 2024-04-26T01:08:14.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.150 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.150 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.150 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.458756+0000", 2024-04-26T01:08:14.150 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011815275 2024-04-26T01:08:14.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.150 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.150 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:14.150 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.459005+0000", 2024-04-26T01:08:14.150 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000249418 2024-04-26T01:08:14.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.150 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.150 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.460129+0000", 2024-04-26T01:08:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001123768 2024-04-26T01:08:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461504+0000", 2024-04-26T01:08:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001375079 2024-04-26T01:08:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.290324+0000", 2024-04-26T01:08:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.82881998700000004 2024-04-26T01:08:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.290524+0000", 2024-04-26T01:08:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000199815 2024-04-26T01:08:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/30 ECSubWriteReply(tid=25, last_complete=37'19, committed=1, applied=1))", 2024-04-26T01:08:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:03.461054+0000", 2024-04-26T01:08:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.619155314, 2024-04-26T01:08:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.83058926899999996, 2024-04-26T01:08:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461054+0000", 2024-04-26T01:08:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461054+0000", 2024-04-26T01:08:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461056+0000", 2024-04-26T01:08:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.906e-06 2024-04-26T01:08:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461069+0000", 2024-04-26T01:08:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.295e-05 2024-04-26T01:08:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.289919+0000", 2024-04-26T01:08:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.82884987300000001 2024-04-26T01:08:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.289933+0000", 2024-04-26T01:08:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4430000000000001e-05 2024-04-26T01:08:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.290082+0000", 2024-04-26T01:08:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014943899999999999 2024-04-26T01:08:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.291643+0000", 2024-04-26T01:08:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001560671 2024-04-26T01:08:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:385 3.fs1 3.db40cadf (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.452224+0000", 2024-04-26T01:08:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.6279855630000002, 2024-04-26T01:08:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95265389199999995, 2024-04-26T01:08:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 385 2024-04-26T01:08:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452224+0000", 2024-04-26T01:08:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452224+0000", 2024-04-26T01:08:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452227+0000", 2024-04-26T01:08:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6830000000000001e-06 2024-04-26T01:08:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452242+0000", 2024-04-26T01:08:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4493e-05 2024-04-26T01:08:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452274+0000", 2024-04-26T01:08:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1806e-05 2024-04-26T01:08:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452300+0000", 2024-04-26T01:08:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6092999999999999e-05 2024-04-26T01:08:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452469+0000", 2024-04-26T01:08:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000169629 2024-04-26T01:08:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:08:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452475+0000", 2024-04-26T01:08:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.604e-06 2024-04-26T01:08:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.404853+0000", 2024-04-26T01:08:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95237828400000002 2024-04-26T01:08:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.404878+0000", 2024-04-26T01:08:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4300000000000001e-05 2024-04-26T01:08:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:386 3.fs1 3.db40cadf (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.452348+0000", 2024-04-26T01:08:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.6278613609999999, 2024-04-26T01:08:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95267636899999997, 2024-04-26T01:08:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 386 2024-04-26T01:08:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452348+0000", 2024-04-26T01:08:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452348+0000", 2024-04-26T01:08:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452349+0000", 2024-04-26T01:08:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.533e-06 2024-04-26T01:08:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452359+0000", 2024-04-26T01:08:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.5070000000000006e-06 2024-04-26T01:08:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452381+0000", 2024-04-26T01:08:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2306e-05 2024-04-26T01:08:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452398+0000", 2024-04-26T01:08:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6456e-05 2024-04-26T01:08:14.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.164 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.164 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.164 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452605+0000", 2024-04-26T01:08:14.164 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00020750900000000001 2024-04-26T01:08:14.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.164 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.164 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:08:14.164 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452610+0000", 2024-04-26T01:08:14.164 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3569999999999997e-06 2024-04-26T01:08:14.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.164 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.164 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.164 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.404988+0000", 2024-04-26T01:08:14.165 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95237776100000004 2024-04-26T01:08:14.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.165 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.165 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.165 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405024+0000", 2024-04-26T01:08:14.165 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5939999999999998e-05 2024-04-26T01:08:14.165 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.165 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.165 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.165 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.165 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:387 3.fs1 3.db40cadf (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:14.165 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.452423+0000", 2024-04-26T01:08:14.165 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.6277859550000002, 2024-04-26T01:08:14.165 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95294480000000004, 2024-04-26T01:08:14.166 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.166 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.166 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.166 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:14.166 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:14.166 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:14.166 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 387 2024-04-26T01:08:14.166 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.166 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.166 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.166 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.166 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452423+0000", 2024-04-26T01:08:14.166 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.166 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.167 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.167 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.167 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452423+0000", 2024-04-26T01:08:14.167 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.167 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.167 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.167 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452425+0000", 2024-04-26T01:08:14.167 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.373e-06 2024-04-26T01:08:14.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.167 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.167 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.167 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452435+0000", 2024-04-26T01:08:14.167 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0149999999999999e-05 2024-04-26T01:08:14.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.168 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.168 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.168 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452447+0000", 2024-04-26T01:08:14.168 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.199e-05 2024-04-26T01:08:14.168 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.168 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.168 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.168 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452461+0000", 2024-04-26T01:08:14.168 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4503e-05 2024-04-26T01:08:14.168 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.168 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.168 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.168 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452685+0000", 2024-04-26T01:08:14.168 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00022367200000000001 2024-04-26T01:08:14.168 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.169 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.169 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:08:14.169 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452687+0000", 2024-04-26T01:08:14.169 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.446e-06 2024-04-26T01:08:14.169 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.169 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.169 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.169 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405319+0000", 2024-04-26T01:08:14.169 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.952631279 2024-04-26T01:08:14.169 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.169 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.169 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.169 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405368+0000", 2024-04-26T01:08:14.170 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9387e-05 2024-04-26T01:08:14.170 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.170 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.170 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.170 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.170 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.170 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:393 3.fs1 3.94f9353f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:14.170 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.464778+0000", 2024-04-26T01:08:14.170 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.6154314269999999, 2024-04-26T01:08:14.170 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94080343600000005, 2024-04-26T01:08:14.170 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.170 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.170 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.170 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:14.171 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:14.171 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:14.171 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 393 2024-04-26T01:08:14.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.171 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.171 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.171 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.171 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464778+0000", 2024-04-26T01:08:14.171 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.171 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.171 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.171 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464778+0000", 2024-04-26T01:08:14.172 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.172 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.172 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.172 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464780+0000", 2024-04-26T01:08:14.172 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2460000000000002e-06 2024-04-26T01:08:14.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.172 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.172 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.172 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464791+0000", 2024-04-26T01:08:14.172 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.145e-05 2024-04-26T01:08:14.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.172 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.172 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.172 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464823+0000", 2024-04-26T01:08:14.173 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1157e-05 2024-04-26T01:08:14.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.173 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.173 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464837+0000", 2024-04-26T01:08:14.173 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.488e-05 2024-04-26T01:08:14.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.173 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.173 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464985+0000", 2024-04-26T01:08:14.173 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000147999 2024-04-26T01:08:14.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.174 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:08:14.174 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464991+0000", 2024-04-26T01:08:14.174 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3329999999999996e-06 2024-04-26T01:08:14.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.174 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.174 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405522+0000", 2024-04-26T01:08:14.174 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94053098899999998 2024-04-26T01:08:14.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.174 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.174 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405581+0000", 2024-04-26T01:08:14.174 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9382000000000002e-05 2024-04-26T01:08:14.175 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.175 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.175 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.175 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.175 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:394 3.fs1 3.94f9353f (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:14.175 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.464870+0000", 2024-04-26T01:08:14.175 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.6153391509999997, 2024-04-26T01:08:14.175 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94122545000000002, 2024-04-26T01:08:14.175 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.175 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.175 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.175 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:14.175 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:14.175 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:14.176 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 394 2024-04-26T01:08:14.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.176 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.176 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.176 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.176 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464870+0000", 2024-04-26T01:08:14.176 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.176 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.176 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.176 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464870+0000", 2024-04-26T01:08:14.176 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464871+0000", 2024-04-26T01:08:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1230000000000001e-06 2024-04-26T01:08:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464875+0000", 2024-04-26T01:08:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.25e-06 2024-04-26T01:08:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464895+0000", 2024-04-26T01:08:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9332999999999999e-05 2024-04-26T01:08:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.178 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.178 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.178 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464904+0000", 2024-04-26T01:08:14.178 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9230000000000008e-06 2024-04-26T01:08:14.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.178 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.178 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.178 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.465051+0000", 2024-04-26T01:08:14.178 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000147186 2024-04-26T01:08:14.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.178 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.178 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:08:14.178 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.465054+0000", 2024-04-26T01:08:14.178 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7530000000000002e-06 2024-04-26T01:08:14.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.179 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.179 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.406060+0000", 2024-04-26T01:08:14.179 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94100597900000005 2024-04-26T01:08:14.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.179 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.179 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.406095+0000", 2024-04-26T01:08:14.179 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5902999999999998e-05 2024-04-26T01:08:14.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.179 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.179 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.180 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:14.181 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:08:14.182 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:14.181+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:14.282 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:14.284+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:14.307 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:14.307 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:08:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:14.322 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:08:14.383 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:14.380+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:14.444 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:14.444 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:14.444 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:08:14.444 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:08:14.444 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:14.458 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:08:14.484 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:14.480+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:14.583 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:08:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:08:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:08:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 33/30 ECSubWrite(tid=1, reqid=client.4521.0:23, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.312551+0000", 2024-04-26T01:08:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.268501848, 2024-04-26T01:08:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0047013630000000001, 2024-04-26T01:08:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312551+0000", 2024-04-26T01:08:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312551+0000", 2024-04-26T01:08:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312552+0000", 2024-04-26T01:08:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1829999999999999e-06 2024-04-26T01:08:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312596+0000", 2024-04-26T01:08:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.4421e-05 2024-04-26T01:08:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312657+0000", 2024-04-26T01:08:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0498000000000003e-05 2024-04-26T01:08:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312670+0000", 2024-04-26T01:08:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3217000000000001e-05 2024-04-26T01:08:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313538+0000", 2024-04-26T01:08:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00086757800000000003 2024-04-26T01:08:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313554+0000", 2024-04-26T01:08:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6385e-05 2024-04-26T01:08:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.317168+0000", 2024-04-26T01:08:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0036138989999999998 2024-04-26T01:08:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.317252+0000", 2024-04-26T01:08:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.4182000000000001e-05 2024-04-26T01:08:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.8s6 32/30 ECSubWrite(tid=1, reqid=client.4521.0:8, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.312770+0000", 2024-04-26T01:08:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.268283007999999, 2024-04-26T01:08:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0045346190000000001, 2024-04-26T01:08:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312770+0000", 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312770+0000", 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312770+0000", 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8000000000000004e-07 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312794+0000", 2024-04-26T01:08:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3346999999999999e-05 2024-04-26T01:08:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312835+0000", 2024-04-26T01:08:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.1659999999999998e-05 2024-04-26T01:08:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312845+0000", 2024-04-26T01:08:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7289999999999995e-06 2024-04-26T01:08:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313322+0000", 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00047711699999999999 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313338+0000", 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5781e-05 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.317267+0000", 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0039284000000000003 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.317304+0000", 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7704999999999999e-05 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs4 32/30 ECSubWrite(tid=1, reqid=client.4521.0:25, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.312890+0000", 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.268162722, 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0042880499999999998, 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312890+0000", 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312890+0000", 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312891+0000", 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.032e-06 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313093+0000", 2024-04-26T01:08:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00020188899999999999 2024-04-26T01:08:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313137+0000", 2024-04-26T01:08:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3701000000000001e-05 2024-04-26T01:08:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313147+0000", 2024-04-26T01:08:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0872e-05 2024-04-26T01:08:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313309+0000", 2024-04-26T01:08:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016174000000000001 2024-04-26T01:08:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313322+0000", 2024-04-26T01:08:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2690999999999999e-05 2024-04-26T01:08:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.317130+0000", 2024-04-26T01:08:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003808437 2024-04-26T01:08:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.317178+0000", 2024-04-26T01:08:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.7688000000000001e-05 2024-04-26T01:08:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs4 32/30 ECSubWrite(tid=3, reqid=client.4521.0:26, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.315812+0000", 2024-04-26T01:08:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.265240858, 2024-04-26T01:08:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0041963729999999998, 2024-04-26T01:08:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315812+0000", 2024-04-26T01:08:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315812+0000", 2024-04-26T01:08:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315813+0000", 2024-04-26T01:08:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.311e-06 2024-04-26T01:08:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315836+0000", 2024-04-26T01:08:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3156999999999999e-05 2024-04-26T01:08:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315879+0000", 2024-04-26T01:08:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3080000000000001e-05 2024-04-26T01:08:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.599 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.599 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315888+0000", 2024-04-26T01:08:14.599 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2509999999999994e-06 2024-04-26T01:08:14.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315944+0000", 2024-04-26T01:08:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.6366999999999998e-05 2024-04-26T01:08:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315951+0000", 2024-04-26T01:08:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7259999999999996e-06 2024-04-26T01:08:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.319970+0000", 2024-04-26T01:08:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0040191970000000004 2024-04-26T01:08:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320008+0000", 2024-04-26T01:08:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8284000000000001e-05 2024-04-26T01:08:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 33/30 ECSubWrite(tid=8, reqid=client.4521.0:37, at_version=33'3, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.324347+0000", 2024-04-26T01:08:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.256705601, 2024-04-26T01:08:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0052585100000000001, 2024-04-26T01:08:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.324347+0000", 2024-04-26T01:08:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.324347+0000", 2024-04-26T01:08:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.324347+0000", 2024-04-26T01:08:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0800000000000001e-07 2024-04-26T01:08:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.324608+0000", 2024-04-26T01:08:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00026080500000000002 2024-04-26T01:08:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.324641+0000", 2024-04-26T01:08:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.3117999999999997e-05 2024-04-26T01:08:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.324648+0000", 2024-04-26T01:08:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5760000000000002e-06 2024-04-26T01:08:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.324908+0000", 2024-04-26T01:08:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00026048500000000002 2024-04-26T01:08:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.324915+0000", 2024-04-26T01:08:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.2400000000000004e-06 2024-04-26T01:08:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.329564+0000", 2024-04-26T01:08:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0046497739999999997 2024-04-26T01:08:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.329606+0000", 2024-04-26T01:08:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.1204000000000001e-05 2024-04-26T01:08:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs4 33/30 ECSubWrite(tid=4, reqid=client.4521.0:26, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.326921+0000", 2024-04-26T01:08:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.254131113, 2024-04-26T01:08:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0053230209999999998, 2024-04-26T01:08:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.326921+0000", 2024-04-26T01:08:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.326921+0000", 2024-04-26T01:08:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.326922+0000", 2024-04-26T01:08:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.8299999999999995e-07 2024-04-26T01:08:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.326951+0000", 2024-04-26T01:08:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8390000000000001e-05 2024-04-26T01:08:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.326995+0000", 2024-04-26T01:08:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.4592999999999998e-05 2024-04-26T01:08:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327006+0000", 2024-04-26T01:08:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0675e-05 2024-04-26T01:08:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327071+0000", 2024-04-26T01:08:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.4912999999999993e-05 2024-04-26T01:08:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327081+0000", 2024-04-26T01:08:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7710000000000001e-06 2024-04-26T01:08:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.332208+0000", 2024-04-26T01:08:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0051267869999999998 2024-04-26T01:08:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.332245+0000", 2024-04-26T01:08:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6909000000000002e-05 2024-04-26T01:08:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.0s5 33/30 ECSubWrite(tid=1, reqid=client.4521.0:104, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.428045+0000", 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.15300779, 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0042507070000000003, 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428045+0000", 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428045+0000", 2024-04-26T01:08:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428046+0000", 2024-04-26T01:08:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.4600000000000003e-07 2024-04-26T01:08:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428069+0000", 2024-04-26T01:08:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3346e-05 2024-04-26T01:08:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428121+0000", 2024-04-26T01:08:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.1688999999999999e-05 2024-04-26T01:08:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428133+0000", 2024-04-26T01:08:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2746e-05 2024-04-26T01:08:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.429365+0000", 2024-04-26T01:08:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0012318870000000001 2024-04-26T01:08:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.429378+0000", 2024-04-26T01:08:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2661e-05 2024-04-26T01:08:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.432264+0000", 2024-04-26T01:08:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0028855170000000002 2024-04-26T01:08:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.432296+0000", 2024-04-26T01:08:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.2014999999999998e-05 2024-04-26T01:08:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 33/30 ECSubWrite(tid=32, reqid=client.4521.0:119, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.448529+0000", 2024-04-26T01:08:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.132523824, 2024-04-26T01:08:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0039279650000000003, 2024-04-26T01:08:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.448529+0000", 2024-04-26T01:08:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.448529+0000", 2024-04-26T01:08:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.448529+0000", 2024-04-26T01:08:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.1099999999999996e-07 2024-04-26T01:08:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.448543+0000", 2024-04-26T01:08:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.402e-05 2024-04-26T01:08:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.449230+0000", 2024-04-26T01:08:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00068687200000000005 2024-04-26T01:08:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.449238+0000", 2024-04-26T01:08:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.1019999999999997e-06 2024-04-26T01:08:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.449334+0000", 2024-04-26T01:08:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6207999999999998e-05 2024-04-26T01:08:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.449345+0000", 2024-04-26T01:08:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0057e-05 2024-04-26T01:08:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.452421+0000", 2024-04-26T01:08:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0030768039999999998 2024-04-26T01:08:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.452457+0000", 2024-04-26T01:08:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5391e-05 2024-04-26T01:08:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 33/30 ECSubWrite(tid=33, reqid=client.4521.0:65, at_version=0'0, trim_to=0'0, roll_forward_to=33'8))", 2024-04-26T01:08:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.448546+0000", 2024-04-26T01:08:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.132506846, 2024-04-26T01:08:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0042872370000000002, 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.448546+0000", 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.448546+0000", 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.448546+0000", 2024-04-26T01:08:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7399999999999999e-07 2024-04-26T01:08:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.448555+0000", 2024-04-26T01:08:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5429999999999994e-06 2024-04-26T01:08:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.449273+0000", 2024-04-26T01:08:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00071857300000000002 2024-04-26T01:08:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.449288+0000", 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4422e-05 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.449444+0000", 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015653499999999999 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.449455+0000", 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0835999999999999e-05 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.452805+0000", 2024-04-26T01:08:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0033497560000000002 2024-04-26T01:08:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.452833+0000", 2024-04-26T01:08:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8297999999999999e-05 2024-04-26T01:08:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 33/30 ECSubWrite(tid=36, reqid=client.4521.0:125, at_version=33'10, trim_to=0'0, roll_forward_to=33'8))", 2024-04-26T01:08:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.453520+0000", 2024-04-26T01:08:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.127532779999999, 2024-04-26T01:08:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0044232150000000003, 2024-04-26T01:08:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.453520+0000", 2024-04-26T01:08:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.453520+0000", 2024-04-26T01:08:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.453521+0000", 2024-04-26T01:08:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.0599999999999999e-07 2024-04-26T01:08:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.453538+0000", 2024-04-26T01:08:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7065999999999999e-05 2024-04-26T01:08:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.453576+0000", 2024-04-26T01:08:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8138e-05 2024-04-26T01:08:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.453584+0000", 2024-04-26T01:08:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.3599999999999996e-06 2024-04-26T01:08:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.453630+0000", 2024-04-26T01:08:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.5507000000000003e-05 2024-04-26T01:08:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.453636+0000", 2024-04-26T01:08:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.8599999999999998e-06 2024-04-26T01:08:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.457906+0000", 2024-04-26T01:08:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0042708549999999996 2024-04-26T01:08:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.457943+0000", 2024-04-26T01:08:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6622999999999998e-05 2024-04-26T01:08:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.0s5 33/30 ECSubWrite(tid=9, reqid=client.4521.0:122, at_version=0'0, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:08:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.455486+0000", 2024-04-26T01:08:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.125566437, 2024-04-26T01:08:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0049709660000000003, 2024-04-26T01:08:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455486+0000", 2024-04-26T01:08:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455486+0000", 2024-04-26T01:08:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455487+0000", 2024-04-26T01:08:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.0400000000000005e-07 2024-04-26T01:08:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455498+0000", 2024-04-26T01:08:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1415e-05 2024-04-26T01:08:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455555+0000", 2024-04-26T01:08:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.6589999999999999e-05 2024-04-26T01:08:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455564+0000", 2024-04-26T01:08:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8459999999999997e-06 2024-04-26T01:08:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455649+0000", 2024-04-26T01:08:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.551e-05 2024-04-26T01:08:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455661+0000", 2024-04-26T01:08:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1406000000000001e-05 2024-04-26T01:08:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.460417+0000", 2024-04-26T01:08:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0047560450000000004 2024-04-26T01:08:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.460457+0000", 2024-04-26T01:08:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.0349999999999998e-05 2024-04-26T01:08:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 33/30 ECSubWrite(tid=37, reqid=client.4521.0:119, at_version=0'0, trim_to=0'0, roll_forward_to=33'2))", 2024-04-26T01:08:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.455627+0000", 2024-04-26T01:08:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.125425366, 2024-04-26T01:08:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0048414199999999999, 2024-04-26T01:08:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455627+0000", 2024-04-26T01:08:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455627+0000", 2024-04-26T01:08:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455628+0000", 2024-04-26T01:08:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.6599999999999995e-07 2024-04-26T01:08:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455638+0000", 2024-04-26T01:08:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.024e-05 2024-04-26T01:08:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455672+0000", 2024-04-26T01:08:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.3263000000000003e-05 2024-04-26T01:08:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455681+0000", 2024-04-26T01:08:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6139999999999998e-06 2024-04-26T01:08:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455783+0000", 2024-04-26T01:08:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000101569 2024-04-26T01:08:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455793+0000", 2024-04-26T01:08:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0040999999999999e-05 2024-04-26T01:08:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.460434+0000", 2024-04-26T01:08:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0046408960000000003 2024-04-26T01:08:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.460469+0000", 2024-04-26T01:08:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5030999999999997e-05 2024-04-26T01:08:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.5s6 33/30 ECSubWrite(tid=9, reqid=client.4521.0:128, at_version=33'6, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:08:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.458110+0000", 2024-04-26T01:08:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.122942269999999, 2024-04-26T01:08:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0048998879999999998, 2024-04-26T01:08:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.458110+0000", 2024-04-26T01:08:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.458110+0000", 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.458111+0000", 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.8299999999999996e-07 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.458125+0000", 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4181e-05 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.458156+0000", 2024-04-26T01:08:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1213000000000001e-05 2024-04-26T01:08:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.458164+0000", 2024-04-26T01:08:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.3889999999999999e-06 2024-04-26T01:08:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.458216+0000", 2024-04-26T01:08:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2101000000000001e-05 2024-04-26T01:08:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.458223+0000", 2024-04-26T01:08:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.0709999999999997e-06 2024-04-26T01:08:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.462983+0000", 2024-04-26T01:08:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0047603560000000003 2024-04-26T01:08:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.463010+0000", 2024-04-26T01:08:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6894e-05 2024-04-26T01:08:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 33/30 ECSubWrite(tid=49, reqid=client.4521.0:154, at_version=33'11, trim_to=0'0, roll_forward_to=33'10))", 2024-04-26T01:08:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.504591+0000", 2024-04-26T01:08:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.076461536, 2024-04-26T01:08:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0042130090000000002, 2024-04-26T01:08:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504591+0000", 2024-04-26T01:08:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504591+0000", 2024-04-26T01:08:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504592+0000", 2024-04-26T01:08:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9999999999999996e-07 2024-04-26T01:08:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504662+0000", 2024-04-26T01:08:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9970999999999998e-05 2024-04-26T01:08:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.506940+0000", 2024-04-26T01:08:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0022778630000000002 2024-04-26T01:08:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.506950+0000", 2024-04-26T01:08:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.057e-05 2024-04-26T01:08:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.507039+0000", 2024-04-26T01:08:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8420999999999999e-05 2024-04-26T01:08:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.507050+0000", 2024-04-26T01:08:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1301000000000001e-05 2024-04-26T01:08:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.508758+0000", 2024-04-26T01:08:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0017083479999999999 2024-04-26T01:08:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.508804+0000", 2024-04-26T01:08:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.5735000000000002e-05 2024-04-26T01:08:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.5s6 33/30 ECSubWrite(tid=10, reqid=client.4521.0:180, at_version=33'7, trim_to=0'0, roll_forward_to=33'5))", 2024-04-26T01:08:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.532467+0000", 2024-04-26T01:08:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.048585472999999, 2024-04-26T01:08:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0040058469999999999, 2024-04-26T01:08:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.532467+0000", 2024-04-26T01:08:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.532467+0000", 2024-04-26T01:08:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.532468+0000", 2024-04-26T01:08:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0869999999999999e-06 2024-04-26T01:08:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.532683+0000", 2024-04-26T01:08:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000215078 2024-04-26T01:08:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.532727+0000", 2024-04-26T01:08:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.4022e-05 2024-04-26T01:08:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.532737+0000", 2024-04-26T01:08:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.4609999999999997e-06 2024-04-26T01:08:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.532793+0000", 2024-04-26T01:08:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.6295999999999998e-05 2024-04-26T01:08:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.532799+0000", 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.1240000000000002e-06 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.536439+0000", 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003639995 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.536473+0000", 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.3784e-05 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.as5 33/30 ECSubWrite(tid=1, reqid=client.4521.0:59, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.567269+0000", 2024-04-26T01:08:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.0137839, 2024-04-26T01:08:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0044497950000000003, 2024-04-26T01:08:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.567269+0000", 2024-04-26T01:08:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.567269+0000", 2024-04-26T01:08:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.567270+0000", 2024-04-26T01:08:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.0800000000000003e-07 2024-04-26T01:08:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.567290+0000", 2024-04-26T01:08:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0732000000000001e-05 2024-04-26T01:08:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.567331+0000", 2024-04-26T01:08:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.0782999999999999e-05 2024-04-26T01:08:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.567339+0000", 2024-04-26T01:08:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.1429999999999998e-06 2024-04-26T01:08:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.567418+0000", 2024-04-26T01:08:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.8885000000000001e-05 2024-04-26T01:08:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.567427+0000", 2024-04-26T01:08:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.0510000000000002e-06 2024-04-26T01:08:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.571670+0000", 2024-04-26T01:08:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0042426629999999998 2024-04-26T01:08:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.571719+0000", 2024-04-26T01:08:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.863e-05 2024-04-26T01:08:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 36/30 ECSubWrite(tid=15, reqid=client.4521.0:233, at_version=36'11, trim_to=0'0, roll_forward_to=33'9))", 2024-04-26T01:08:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:02.480147+0000", 2024-04-26T01:08:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 12.100905449000001, 2024-04-26T01:08:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.64949222799999995, 2024-04-26T01:08:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480147+0000", 2024-04-26T01:08:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480147+0000", 2024-04-26T01:08:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480148+0000", 2024-04-26T01:08:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2300000000000001e-06 2024-04-26T01:08:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480517+0000", 2024-04-26T01:08:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00036900700000000003 2024-04-26T01:08:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125262+0000", 2024-04-26T01:08:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.64474460600000005 2024-04-26T01:08:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125278+0000", 2024-04-26T01:08:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5622999999999999e-05 2024-04-26T01:08:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125483+0000", 2024-04-26T01:08:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000205105 2024-04-26T01:08:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125502+0000", 2024-04-26T01:08:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8876999999999999e-05 2024-04-26T01:08:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.129565+0000", 2024-04-26T01:08:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0040633930000000002 2024-04-26T01:08:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.129639+0000", 2024-04-26T01:08:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4387000000000005e-05 2024-04-26T01:08:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.5s6 37/30 ECSubWrite(tid=21, reqid=client.4521.0:250, at_version=37'16, trim_to=0'0, roll_forward_to=36'13))", 2024-04-26T01:08:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:03.432237+0000", 2024-04-26T01:08:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.148816031000001, 2024-04-26T01:08:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0046784119999999998, 2024-04-26T01:08:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.432237+0000", 2024-04-26T01:08:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.432237+0000", 2024-04-26T01:08:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.432238+0000", 2024-04-26T01:08:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2330000000000001e-06 2024-04-26T01:08:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.432310+0000", 2024-04-26T01:08:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2136000000000002e-05 2024-04-26T01:08:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.432450+0000", 2024-04-26T01:08:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014023900000000001 2024-04-26T01:08:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.432471+0000", 2024-04-26T01:08:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1046000000000001e-05 2024-04-26T01:08:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.432651+0000", 2024-04-26T01:08:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017933599999999999 2024-04-26T01:08:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.432665+0000", 2024-04-26T01:08:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4423000000000001e-05 2024-04-26T01:08:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.436786+0000", 2024-04-26T01:08:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0041206469999999999 2024-04-26T01:08:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.436915+0000", 2024-04-26T01:08:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000129352 2024-04-26T01:08:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.5s6 37/30 ECSubWrite(tid=22, reqid=client.4521.0:251, at_version=37'17, trim_to=0'0, roll_forward_to=36'13))", 2024-04-26T01:08:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:03.434840+0000", 2024-04-26T01:08:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.146212316, 2024-04-26T01:08:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0044784309999999997, 2024-04-26T01:08:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.434840+0000", 2024-04-26T01:08:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.434840+0000", 2024-04-26T01:08:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.434842+0000", 2024-04-26T01:08:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4100000000000001e-06 2024-04-26T01:08:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.434884+0000", 2024-04-26T01:08:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.2453999999999997e-05 2024-04-26T01:08:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.434972+0000", 2024-04-26T01:08:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8299000000000003e-05 2024-04-26T01:08:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.434997+0000", 2024-04-26T01:08:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4369999999999999e-05 2024-04-26T01:08:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.435120+0000", 2024-04-26T01:08:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012300899999999999 2024-04-26T01:08:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.435135+0000", 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.539e-05 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.439240+0000", 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0041043829999999996 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.439319+0000", 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9116000000000006e-05 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.5s6 43/30 ECSubWrite(tid=35, reqid=osd.15.0:35, at_version=43'30, trim_to=0'0, roll_forward_to=42'28))", 2024-04-26T01:08:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 5.1744937090000001, 2024-04-26T01:08:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011310863000000001, 2024-04-26T01:08:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2e-07 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406612+0000", 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2206000000000001e-05 2024-04-26T01:08:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406709+0000", 2024-04-26T01:08:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7106000000000002e-05 2024-04-26T01:08:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406737+0000", 2024-04-26T01:08:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8629999999999999e-05 2024-04-26T01:08:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.407351+0000", 2024-04-26T01:08:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00061410800000000004 2024-04-26T01:08:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.407377+0000", 2024-04-26T01:08:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5057000000000001e-05 2024-04-26T01:08:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.417771+0000", 2024-04-26T01:08:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.01039402 2024-04-26T01:08:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.417870+0000", 2024-04-26T01:08:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9215999999999993e-05 2024-04-26T01:08:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:14.674 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:14.675 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:08:14.677 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:14.672+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:14.776 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:14.776+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:14.848 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:08:14.849 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:14.863 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:08:14.912 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:14.912+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:14.978 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:14.973+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:14.986 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:14.987 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:14.987 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:08:14.987 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:08:14.987 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:15.000 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:08:15.040 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-04-26T01:08:15.045 INFO:tasks.rados.rados.0.smithi164.stdout:188: done (1 left) 2024-04-26T01:08:15.046 INFO:tasks.rados.rados.0.smithi164.stdout:189: done (0 left) 2024-04-26T01:08:15.046 INFO:tasks.rados.rados.0.smithi164.stdout:190: snap_remove snap 6 2024-04-26T01:08:15.114 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:15.112+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:15.158 INFO:teuthology.orchestra.run.smithi164.stdout: marking collection for removal 2024-04-26T01:08:15.158 INFO:teuthology.orchestra.run.smithi164.stdout:setting '_remove' omap key 2024-04-26T01:08:15.158 INFO:teuthology.orchestra.run.smithi164.stdout:finish_remove_pgs 3.as6_head removing 3.as6 2024-04-26T01:08:15.164 INFO:teuthology.orchestra.run.smithi164.stdout:Remove successful 2024-04-26T01:08:15.171 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:08:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:08:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:08:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 32/30 ECSubWrite(tid=0, reqid=client.4521.0:10, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.308918+0000", 2024-04-26T01:08:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.859814437000001, 2024-04-26T01:08:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0036702219999999999, 2024-04-26T01:08:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.308918+0000", 2024-04-26T01:08:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.308918+0000", 2024-04-26T01:08:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.308919+0000", 2024-04-26T01:08:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9799999999999999e-07 2024-04-26T01:08:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.308966+0000", 2024-04-26T01:08:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.7073000000000001e-05 2024-04-26T01:08:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309034+0000", 2024-04-26T01:08:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.8249000000000003e-05 2024-04-26T01:08:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309047+0000", 2024-04-26T01:08:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3502e-05 2024-04-26T01:08:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309199+0000", 2024-04-26T01:08:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015163500000000001 2024-04-26T01:08:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309212+0000", 2024-04-26T01:08:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2928999999999999e-05 2024-04-26T01:08:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312539+0000", 2024-04-26T01:08:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003326655 2024-04-26T01:08:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312589+0000", 2024-04-26T01:08:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9880999999999999e-05 2024-04-26T01:08:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 32/30 ECSubWrite(tid=1, reqid=client.4521.0:25, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.854900201, 2024-04-26T01:08:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062762640000000001, 2024-04-26T01:08:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2700000000000001e-07 2024-04-26T01:08:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313923+0000", 2024-04-26T01:08:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.0530000000000002e-05 2024-04-26T01:08:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314064+0000", 2024-04-26T01:08:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000140841 2024-04-26T01:08:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314070+0000", 2024-04-26T01:08:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.976e-06 2024-04-26T01:08:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314140+0000", 2024-04-26T01:08:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9664000000000003e-05 2024-04-26T01:08:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314151+0000", 2024-04-26T01:08:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1251e-05 2024-04-26T01:08:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320065+0000", 2024-04-26T01:08:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059141990000000002 2024-04-26T01:08:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320109+0000", 2024-04-26T01:08:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3575999999999997e-05 2024-04-26T01:08:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.ds3 33/30 ECSubWrite(tid=2, reqid=client.4521.0:5, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.315545+0000", 2024-04-26T01:08:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.853187236, 2024-04-26T01:08:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0045609129999999998, 2024-04-26T01:08:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315545+0000", 2024-04-26T01:08:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315545+0000", 2024-04-26T01:08:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315547+0000", 2024-04-26T01:08:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.096e-06 2024-04-26T01:08:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315567+0000", 2024-04-26T01:08:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0075e-05 2024-04-26T01:08:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315620+0000", 2024-04-26T01:08:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3238000000000001e-05 2024-04-26T01:08:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315633+0000", 2024-04-26T01:08:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3379e-05 2024-04-26T01:08:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315695+0000", 2024-04-26T01:08:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.1875e-05 2024-04-26T01:08:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.315704+0000", 2024-04-26T01:08:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.3000000000000007e-06 2024-04-26T01:08:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320068+0000", 2024-04-26T01:08:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0043636040000000001 2024-04-26T01:08:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320106+0000", 2024-04-26T01:08:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8346000000000002e-05 2024-04-26T01:08:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 32/30 ECSubWrite(tid=3, reqid=client.4521.0:26, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.316902+0000", 2024-04-26T01:08:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.851830702000001, 2024-04-26T01:08:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0057492339999999998, 2024-04-26T01:08:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.316902+0000", 2024-04-26T01:08:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.316902+0000", 2024-04-26T01:08:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.316903+0000", 2024-04-26T01:08:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2309999999999999e-06 2024-04-26T01:08:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.316927+0000", 2024-04-26T01:08:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4091e-05 2024-04-26T01:08:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.316974+0000", 2024-04-26T01:08:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.6496999999999999e-05 2024-04-26T01:08:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.316986+0000", 2024-04-26T01:08:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1953000000000001e-05 2024-04-26T01:08:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.317067+0000", 2024-04-26T01:08:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.1122e-05 2024-04-26T01:08:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.317079+0000", 2024-04-26T01:08:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2517000000000001e-05 2024-04-26T01:08:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.322622+0000", 2024-04-26T01:08:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00554216 2024-04-26T01:08:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.322651+0000", 2024-04-26T01:08:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9663e-05 2024-04-26T01:08:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.1s4 33/30 ECSubWrite(tid=4, reqid=client.4521.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.320780+0000", 2024-04-26T01:08:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.847952847, 2024-04-26T01:08:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0046243480000000003, 2024-04-26T01:08:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320780+0000", 2024-04-26T01:08:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320780+0000", 2024-04-26T01:08:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320781+0000", 2024-04-26T01:08:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.8700000000000004e-07 2024-04-26T01:08:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320794+0000", 2024-04-26T01:08:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3006e-05 2024-04-26T01:08:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320836+0000", 2024-04-26T01:08:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.2166000000000003e-05 2024-04-26T01:08:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320846+0000", 2024-04-26T01:08:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6099999999999995e-06 2024-04-26T01:08:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320930+0000", 2024-04-26T01:08:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.3857999999999996e-05 2024-04-26T01:08:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320944+0000", 2024-04-26T01:08:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4749999999999999e-05 2024-04-26T01:08:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325361+0000", 2024-04-26T01:08:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0044163630000000004 2024-04-26T01:08:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325404+0000", 2024-04-26T01:08:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3608e-05 2024-04-26T01:08:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.ds3 33/30 ECSubWrite(tid=4, reqid=client.4521.0:34, at_version=33'3, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.323112+0000", 2024-04-26T01:08:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.845620436000001, 2024-04-26T01:08:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0050787269999999999, 2024-04-26T01:08:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.323112+0000", 2024-04-26T01:08:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.323112+0000", 2024-04-26T01:08:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.323113+0000", 2024-04-26T01:08:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7100000000000011e-07 2024-04-26T01:08:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.323273+0000", 2024-04-26T01:08:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000159276 2024-04-26T01:08:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.323361+0000", 2024-04-26T01:08:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8548999999999995e-05 2024-04-26T01:08:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.323384+0000", 2024-04-26T01:08:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2850999999999999e-05 2024-04-26T01:08:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.323460+0000", 2024-04-26T01:08:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5624999999999995e-05 2024-04-26T01:08:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.323470+0000", 2024-04-26T01:08:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0832999999999999e-05 2024-04-26T01:08:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.328154+0000", 2024-04-26T01:08:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0046835749999999997 2024-04-26T01:08:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.328191+0000", 2024-04-26T01:08:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7047e-05 2024-04-26T01:08:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.ds3 33/30 ECSubWrite(tid=5, reqid=client.4521.0:35, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.325729+0000", 2024-04-26T01:08:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.843003541, 2024-04-26T01:08:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079681170000000003, 2024-04-26T01:08:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325729+0000", 2024-04-26T01:08:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325729+0000", 2024-04-26T01:08:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325730+0000", 2024-04-26T01:08:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.78e-07 2024-04-26T01:08:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325748+0000", 2024-04-26T01:08:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8127000000000001e-05 2024-04-26T01:08:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325789+0000", 2024-04-26T01:08:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.1100000000000003e-05 2024-04-26T01:08:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325800+0000", 2024-04-26T01:08:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0256e-05 2024-04-26T01:08:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325859+0000", 2024-04-26T01:08:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9849999999999998e-05 2024-04-26T01:08:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325869+0000", 2024-04-26T01:08:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6989999999999998e-06 2024-04-26T01:08:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.333645+0000", 2024-04-26T01:08:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0077756800000000001 2024-04-26T01:08:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.333697+0000", 2024-04-26T01:08:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2426999999999998e-05 2024-04-26T01:08:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 33/30 ECSubWrite(tid=4, reqid=client.4521.0:26, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.327582+0000", 2024-04-26T01:08:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.841150901000001, 2024-04-26T01:08:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0074218790000000001, 2024-04-26T01:08:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327582+0000", 2024-04-26T01:08:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327582+0000", 2024-04-26T01:08:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327583+0000", 2024-04-26T01:08:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0669999999999999e-06 2024-04-26T01:08:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327596+0000", 2024-04-26T01:08:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3135e-05 2024-04-26T01:08:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327649+0000", 2024-04-26T01:08:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2704999999999998e-05 2024-04-26T01:08:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327659+0000", 2024-04-26T01:08:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.8470000000000008e-06 2024-04-26T01:08:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327733+0000", 2024-04-26T01:08:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4114000000000002e-05 2024-04-26T01:08:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327746+0000", 2024-04-26T01:08:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3176999999999999e-05 2024-04-26T01:08:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.334969+0000", 2024-04-26T01:08:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0072232160000000002 2024-04-26T01:08:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.335004+0000", 2024-04-26T01:08:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4618e-05 2024-04-26T01:08:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpRead(3.cs3 33/30 ECSubRead(tid=5, to_read={3:31be42eb:::smithi16428812-9:head=0,4096,0}, subchunks={3:31be42eb:::smithi16428812-9:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.327601+0000", 2024-04-26T01:08:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.841131732999999, 2024-04-26T01:08:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003394645, 2024-04-26T01:08:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327601+0000", 2024-04-26T01:08:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327601+0000", 2024-04-26T01:08:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327601+0000", 2024-04-26T01:08:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1600000000000002e-07 2024-04-26T01:08:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327608+0000", 2024-04-26T01:08:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.1169999999999998e-06 2024-04-26T01:08:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327688+0000", 2024-04-26T01:08:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9221999999999995e-05 2024-04-26T01:08:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327696+0000", 2024-04-26T01:08:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.388e-06 2024-04-26T01:08:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327973+0000", 2024-04-26T01:08:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00027733599999999999 2024-04-26T01:08:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.330996+0000", 2024-04-26T01:08:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003022266 2024-04-26T01:08:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.ds3 33/30 ECSubWrite(tid=9, reqid=client.4521.0:47, at_version=33'6, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:08:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.345483+0000", 2024-04-26T01:08:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.823249949000001, 2024-04-26T01:08:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0038652399999999998, 2024-04-26T01:08:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.345483+0000", 2024-04-26T01:08:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.345483+0000", 2024-04-26T01:08:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.345484+0000", 2024-04-26T01:08:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5799999999999998e-07 2024-04-26T01:08:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.345501+0000", 2024-04-26T01:08:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7467999999999999e-05 2024-04-26T01:08:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.345536+0000", 2024-04-26T01:08:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5462e-05 2024-04-26T01:08:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.345552+0000", 2024-04-26T01:08:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5047e-05 2024-04-26T01:08:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.345609+0000", 2024-04-26T01:08:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7503999999999997e-05 2024-04-26T01:08:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.345615+0000", 2024-04-26T01:08:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0170000000000002e-06 2024-04-26T01:08:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.349300+0000", 2024-04-26T01:08:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0036846729999999999 2024-04-26T01:08:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.349348+0000", 2024-04-26T01:08:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.8310999999999999e-05 2024-04-26T01:08:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.ds3 33/30 ECSubWrite(tid=10, reqid=client.4521.0:47, at_version=0'0, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:08:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.350352+0000", 2024-04-26T01:08:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.818380548, 2024-04-26T01:08:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003692533, 2024-04-26T01:08:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350352+0000", 2024-04-26T01:08:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350352+0000", 2024-04-26T01:08:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350353+0000", 2024-04-26T01:08:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.065e-06 2024-04-26T01:08:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350364+0000", 2024-04-26T01:08:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0998999999999999e-05 2024-04-26T01:08:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350397+0000", 2024-04-26T01:08:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.2670999999999999e-05 2024-04-26T01:08:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350405+0000", 2024-04-26T01:08:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.6490000000000008e-06 2024-04-26T01:08:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350477+0000", 2024-04-26T01:08:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2412999999999999e-05 2024-04-26T01:08:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350491+0000", 2024-04-26T01:08:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3702e-05 2024-04-26T01:08:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.354004+0000", 2024-04-26T01:08:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003512842 2024-04-26T01:08:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.354045+0000", 2024-04-26T01:08:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.1192000000000001e-05 2024-04-26T01:08:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 33/30 ECSubWrite(tid=12, reqid=client.4521.0:52, at_version=33'3, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.350607+0000", 2024-04-26T01:08:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.81812596, 2024-04-26T01:08:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0037866699999999998, 2024-04-26T01:08:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350607+0000", 2024-04-26T01:08:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350607+0000", 2024-04-26T01:08:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350608+0000", 2024-04-26T01:08:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.6799999999999999e-07 2024-04-26T01:08:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350721+0000", 2024-04-26T01:08:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00011348799999999999 2024-04-26T01:08:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350756+0000", 2024-04-26T01:08:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4714e-05 2024-04-26T01:08:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350767+0000", 2024-04-26T01:08:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1430000000000001e-05 2024-04-26T01:08:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350828+0000", 2024-04-26T01:08:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0971000000000003e-05 2024-04-26T01:08:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.350836+0000", 2024-04-26T01:08:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4170000000000003e-06 2024-04-26T01:08:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.354355+0000", 2024-04-26T01:08:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0035191659999999998 2024-04-26T01:08:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.354393+0000", 2024-04-26T01:08:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8716e-05 2024-04-26T01:08:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 33/30 ECSubWrite(tid=31, reqid=client.4521.0:133, at_version=33'7, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:08:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.466095+0000", 2024-04-26T01:08:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.702638048000001, 2024-04-26T01:08:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0034118030000000001, 2024-04-26T01:08:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.466095+0000", 2024-04-26T01:08:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.466095+0000", 2024-04-26T01:08:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.466096+0000", 2024-04-26T01:08:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4160000000000001e-06 2024-04-26T01:08:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.466586+0000", 2024-04-26T01:08:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00049011200000000001 2024-04-26T01:08:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.466643+0000", 2024-04-26T01:08:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7036000000000001e-05 2024-04-26T01:08:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.466657+0000", 2024-04-26T01:08:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4079e-05 2024-04-26T01:08:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.466740+0000", 2024-04-26T01:08:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2195999999999997e-05 2024-04-26T01:08:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.466752+0000", 2024-04-26T01:08:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.226e-05 2024-04-26T01:08:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.469460+0000", 2024-04-26T01:08:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0027085009999999999 2024-04-26T01:08:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.469506+0000", 2024-04-26T01:08:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.6202999999999998e-05 2024-04-26T01:08:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 33/30 ECSubWrite(tid=32, reqid=client.4521.0:134, at_version=33'8, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:08:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.467026+0000", 2024-04-26T01:08:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.701707128000001, 2024-04-26T01:08:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0055713610000000004, 2024-04-26T01:08:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467026+0000", 2024-04-26T01:08:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467026+0000", 2024-04-26T01:08:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467026+0000", 2024-04-26T01:08:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.6600000000000005e-07 2024-04-26T01:08:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467097+0000", 2024-04-26T01:08:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.0048000000000004e-05 2024-04-26T01:08:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467143+0000", 2024-04-26T01:08:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.6139e-05 2024-04-26T01:08:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467155+0000", 2024-04-26T01:08:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2123000000000001e-05 2024-04-26T01:08:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467373+0000", 2024-04-26T01:08:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00021788499999999999 2024-04-26T01:08:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467384+0000", 2024-04-26T01:08:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0856000000000001e-05 2024-04-26T01:08:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472554+0000", 2024-04-26T01:08:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0051708830000000002 2024-04-26T01:08:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472597+0000", 2024-04-26T01:08:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.2561000000000001e-05 2024-04-26T01:08:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.1s4 33/30 ECSubWrite(tid=54, reqid=client.4521.0:174, at_version=0'0, trim_to=0'0, roll_forward_to=33'11))", 2024-04-26T01:08:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.525625+0000", 2024-04-26T01:08:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.643107973999999, 2024-04-26T01:08:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.004818793, 2024-04-26T01:08:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.525625+0000", 2024-04-26T01:08:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.525625+0000", 2024-04-26T01:08:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.525626+0000", 2024-04-26T01:08:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6200000000000006e-07 2024-04-26T01:08:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.525647+0000", 2024-04-26T01:08:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1693999999999999e-05 2024-04-26T01:08:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.525687+0000", 2024-04-26T01:08:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.9972999999999998e-05 2024-04-26T01:08:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.525697+0000", 2024-04-26T01:08:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.3959999999999995e-06 2024-04-26T01:08:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.525769+0000", 2024-04-26T01:08:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.1895000000000002e-05 2024-04-26T01:08:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.525782+0000", 2024-04-26T01:08:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2931e-05 2024-04-26T01:08:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.530408+0000", 2024-04-26T01:08:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0046260420000000003 2024-04-26T01:08:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.530444+0000", 2024-04-26T01:08:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5899999999999998e-05 2024-04-26T01:08:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.1s4 37/30 ECSubWrite(tid=77, reqid=client.4521.0:258, at_version=37'13, trim_to=0'0, roll_forward_to=33'11))", 2024-04-26T01:08:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:03.435738+0000", 2024-04-26T01:08:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.732995209, 2024-04-26T01:08:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003836065, 2024-04-26T01:08:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.435738+0000", 2024-04-26T01:08:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.435738+0000", 2024-04-26T01:08:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.435738+0000", 2024-04-26T01:08:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7299999999999996e-07 2024-04-26T01:08:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.436048+0000", 2024-04-26T01:08:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000310124 2024-04-26T01:08:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.436133+0000", 2024-04-26T01:08:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5096e-05 2024-04-26T01:08:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.436159+0000", 2024-04-26T01:08:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5372999999999999e-05 2024-04-26T01:08:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.436331+0000", 2024-04-26T01:08:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017257500000000001 2024-04-26T01:08:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.436373+0000", 2024-04-26T01:08:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.1257000000000002e-05 2024-04-26T01:08:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.439477+0000", 2024-04-26T01:08:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0031045479999999999 2024-04-26T01:08:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.439574+0000", 2024-04-26T01:08:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6519000000000002e-05 2024-04-26T01:08:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 37/30 ECSubWrite(tid=65, reqid=client.4521.0:247, at_version=37'18, trim_to=0'0, roll_forward_to=36'16))", 2024-04-26T01:08:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:03.441827+0000", 2024-04-26T01:08:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.726906048, 2024-04-26T01:08:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0040076579999999999, 2024-04-26T01:08:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.441827+0000", 2024-04-26T01:08:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.441827+0000", 2024-04-26T01:08:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.441828+0000", 2024-04-26T01:08:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5799999999999999e-06 2024-04-26T01:08:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.442385+0000", 2024-04-26T01:08:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00055710899999999997 2024-04-26T01:08:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.442451+0000", 2024-04-26T01:08:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5196000000000004e-05 2024-04-26T01:08:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.442466+0000", 2024-04-26T01:08:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.588e-05 2024-04-26T01:08:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.442589+0000", 2024-04-26T01:08:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012304899999999999 2024-04-26T01:08:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.442619+0000", 2024-04-26T01:08:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9145999999999998e-05 2024-04-26T01:08:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.445763+0000", 2024-04-26T01:08:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0031441720000000002 2024-04-26T01:08:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.445834+0000", 2024-04-26T01:08:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.1526000000000006e-05 2024-04-26T01:08:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 37/30 ECSubWrite(tid=66, reqid=client.4521.0:247, at_version=0'0, trim_to=0'0, roll_forward_to=37'18))", 2024-04-26T01:08:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:03.448733+0000", 2024-04-26T01:08:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.719999807000001, 2024-04-26T01:08:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0039072430000000003, 2024-04-26T01:08:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.448733+0000", 2024-04-26T01:08:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.448733+0000", 2024-04-26T01:08:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.448735+0000", 2024-04-26T01:08:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0200000000000001e-06 2024-04-26T01:08:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.448759+0000", 2024-04-26T01:08:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4409999999999998e-05 2024-04-26T01:08:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.448829+0000", 2024-04-26T01:08:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9956000000000006e-05 2024-04-26T01:08:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.448845+0000", 2024-04-26T01:08:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6143000000000001e-05 2024-04-26T01:08:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.448991+0000", 2024-04-26T01:08:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014517299999999999 2024-04-26T01:08:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.449015+0000", 2024-04-26T01:08:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4236e-05 2024-04-26T01:08:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.452546+0000", 2024-04-26T01:08:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0035308190000000001 2024-04-26T01:08:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.452640+0000", 2024-04-26T01:08:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.4486000000000003e-05 2024-04-26T01:08:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.1s4 38/30 ECSubWrite(tid=85, reqid=osd.4.0:85, at_version=38'19, trim_to=0'0, roll_forward_to=38'15))", 2024-04-26T01:08:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:04.432571+0000", 2024-04-26T01:08:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.736162134000001, 2024-04-26T01:08:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003432465, 2024-04-26T01:08:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.432571+0000", 2024-04-26T01:08:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.432571+0000", 2024-04-26T01:08:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.432573+0000", 2024-04-26T01:08:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1560000000000002e-06 2024-04-26T01:08:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.432633+0000", 2024-04-26T01:08:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0080000000000001e-05 2024-04-26T01:08:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.432750+0000", 2024-04-26T01:08:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000117382 2024-04-26T01:08:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.432779+0000", 2024-04-26T01:08:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8710000000000001e-05 2024-04-26T01:08:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.433375+0000", 2024-04-26T01:08:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00059605499999999996 2024-04-26T01:08:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.433404+0000", 2024-04-26T01:08:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8866000000000001e-05 2024-04-26T01:08:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.435907+0000", 2024-04-26T01:08:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0025028339999999998 2024-04-26T01:08:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.436003+0000", 2024-04-26T01:08:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6382e-05 2024-04-26T01:08:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.1s4 38/30 ECSubWrite(tid=86, reqid=client.4521.0:267, at_version=38'21, trim_to=0'0, roll_forward_to=38'15))", 2024-04-26T01:08:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:04.441553+0000", 2024-04-26T01:08:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.727180203, 2024-04-26T01:08:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0040707950000000003, 2024-04-26T01:08:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.441553+0000", 2024-04-26T01:08:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.441553+0000", 2024-04-26T01:08:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.441572+0000", 2024-04-26T01:08:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9817e-05 2024-04-26T01:08:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.442181+0000", 2024-04-26T01:08:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00060864800000000004 2024-04-26T01:08:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.442260+0000", 2024-04-26T01:08:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9042999999999995e-05 2024-04-26T01:08:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.442276+0000", 2024-04-26T01:08:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5716e-05 2024-04-26T01:08:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.442433+0000", 2024-04-26T01:08:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015734599999999999 2024-04-26T01:08:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.442463+0000", 2024-04-26T01:08:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0173000000000001e-05 2024-04-26T01:08:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.445517+0000", 2024-04-26T01:08:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0030541359999999998 2024-04-26T01:08:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.445623+0000", 2024-04-26T01:08:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010591599999999999 2024-04-26T01:08:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.262 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:15.264 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:08:15.264 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:15.264+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:15.364 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:15.364+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:15.434 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:08:15.435 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:15.449 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:08:15.465 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:15.464+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:15.566 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:15.564+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:15.616 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:08:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:08:15.616 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:15.630 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:08:15.672 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:15.672+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:15.694 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-11 --journal-path /var/lib/ceph/osd/ceph-11/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.as6.11 2024-04-26T01:08:15.767 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:15.761+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:15.802 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:08:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:08:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:08:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:103 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~573440,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.424398+0000", 2024-04-26T01:08:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 16.375626555, 2024-04-26T01:08:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012920365, 2024-04-26T01:08:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 103 2024-04-26T01:08:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.424398+0000", 2024-04-26T01:08:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.424398+0000", 2024-04-26T01:08:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.424401+0000", 2024-04-26T01:08:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0680000000000001e-06 2024-04-26T01:08:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425367+0000", 2024-04-26T01:08:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00096614400000000005 2024-04-26T01:08:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425393+0000", 2024-04-26T01:08:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.55e-05 2024-04-26T01:08:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425415+0000", 2024-04-26T01:08:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1959000000000001e-05 2024-04-26T01:08:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425510+0000", 2024-04-26T01:08:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.5134000000000001e-05 2024-04-26T01:08:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425620+0000", 2024-04-26T01:08:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00011056 2024-04-26T01:08:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.426564+0000", 2024-04-26T01:08:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00094343200000000002 2024-04-26T01:08:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428445+0000", 2024-04-26T01:08:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0018816569999999999 2024-04-26T01:08:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437169+0000", 2024-04-26T01:08:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0087233220000000004 2024-04-26T01:08:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437318+0000", 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014958899999999999 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:104 3.0s0 3:0982979f:::smithi16428812-35:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.425460+0000", 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 16.374564713000002, 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012253794, 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 104 2024-04-26T01:08:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425460+0000", 2024-04-26T01:08:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425460+0000", 2024-04-26T01:08:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425461+0000", 2024-04-26T01:08:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7519999999999999e-06 2024-04-26T01:08:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425470+0000", 2024-04-26T01:08:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7029999999999995e-06 2024-04-26T01:08:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425485+0000", 2024-04-26T01:08:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.472e-05 2024-04-26T01:08:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425499+0000", 2024-04-26T01:08:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4228e-05 2024-04-26T01:08:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.427415+0000", 2024-04-26T01:08:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001915618 2024-04-26T01:08:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.812 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.812 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.427541+0000", 2024-04-26T01:08:15.812 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012602699999999999 2024-04-26T01:08:15.812 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.812 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.427948+0000", 2024-04-26T01:08:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00040687999999999998 2024-04-26T01:08:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428977+0000", 2024-04-26T01:08:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0010294119999999999 2024-04-26T01:08:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437611+0000", 2024-04-26T01:08:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0086342789999999999 2024-04-26T01:08:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437714+0000", 2024-04-26T01:08:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000102175 2024-04-26T01:08:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:105 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 16.374504315999999, 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017139178000000001, 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 105 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:08:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:08:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425522+0000", 2024-04-26T01:08:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5430000000000001e-06 2024-04-26T01:08:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425529+0000", 2024-04-26T01:08:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2570000000000001e-06 2024-04-26T01:08:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425548+0000", 2024-04-26T01:08:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8674000000000001e-05 2024-04-26T01:08:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425560+0000", 2024-04-26T01:08:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2683e-05 2024-04-26T01:08:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428152+0000", 2024-04-26T01:08:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0025912800000000001 2024-04-26T01:08:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428234+0000", 2024-04-26T01:08:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2105000000000001e-05 2024-04-26T01:08:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438095+0000", 2024-04-26T01:08:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0098614180000000003 2024-04-26T01:08:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438205+0000", 2024-04-26T01:08:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000109849 2024-04-26T01:08:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442659+0000", 2024-04-26T01:08:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0044543689999999997 2024-04-26T01:08:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:115 3.0s0 3:00e538ad:::smithi16428812-39:head [append 0~638976,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.437822+0000", 2024-04-26T01:08:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 16.362201980999998, 2024-04-26T01:08:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.013307608, 2024-04-26T01:08:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 115 2024-04-26T01:08:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437822+0000", 2024-04-26T01:08:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437822+0000", 2024-04-26T01:08:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437824+0000", 2024-04-26T01:08:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.897e-06 2024-04-26T01:08:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438562+0000", 2024-04-26T01:08:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00073750400000000005 2024-04-26T01:08:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438572+0000", 2024-04-26T01:08:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0570999999999999e-05 2024-04-26T01:08:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438580+0000", 2024-04-26T01:08:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.8809999999999996e-06 2024-04-26T01:08:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.439078+0000", 2024-04-26T01:08:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00049728399999999996 2024-04-26T01:08:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.439233+0000", 2024-04-26T01:08:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000155134 2024-04-26T01:08:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.440207+0000", 2024-04-26T01:08:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00097409999999999999 2024-04-26T01:08:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442108+0000", 2024-04-26T01:08:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0019014139999999999 2024-04-26T01:08:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451061+0000", 2024-04-26T01:08:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0089528560000000004 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451130+0000", 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.8967000000000006e-05 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:116 3.0s0 3:00e538ad:::smithi16428812-39:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.438595+0000", 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 16.361429081000001, 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012771675999999999, 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 116 2024-04-26T01:08:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438595+0000", 2024-04-26T01:08:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438595+0000", 2024-04-26T01:08:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438596+0000", 2024-04-26T01:08:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7400000000000002e-07 2024-04-26T01:08:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438600+0000", 2024-04-26T01:08:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8039999999999999e-06 2024-04-26T01:08:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438607+0000", 2024-04-26T01:08:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.17e-06 2024-04-26T01:08:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438613+0000", 2024-04-26T01:08:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0549999999999996e-06 2024-04-26T01:08:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441123+0000", 2024-04-26T01:08:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0025094510000000002 2024-04-26T01:08:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441245+0000", 2024-04-26T01:08:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012231399999999999 2024-04-26T01:08:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441736+0000", 2024-04-26T01:08:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00049087600000000001 2024-04-26T01:08:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.444541+0000", 2024-04-26T01:08:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0028048249999999999 2024-04-26T01:08:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451266+0000", 2024-04-26T01:08:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0067249800000000002 2024-04-26T01:08:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451367+0000", 2024-04-26T01:08:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000101327 2024-04-26T01:08:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:117 3.0s0 3:00e538ad:::smithi16428812-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:08:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 16.361401319999999, 2024-04-26T01:08:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017480893000000001, 2024-04-26T01:08:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 117 2024-04-26T01:08:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:08:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:08:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438624+0000", 2024-04-26T01:08:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.6000000000000003e-07 2024-04-26T01:08:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438628+0000", 2024-04-26T01:08:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8029999999999998e-06 2024-04-26T01:08:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438635+0000", 2024-04-26T01:08:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2799999999999998e-06 2024-04-26T01:08:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438641+0000", 2024-04-26T01:08:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5459999999999997e-06 2024-04-26T01:08:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442203+0000", 2024-04-26T01:08:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003561594 2024-04-26T01:08:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442287+0000", 2024-04-26T01:08:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.4443000000000004e-05 2024-04-26T01:08:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451430+0000", 2024-04-26T01:08:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0091427130000000002 2024-04-26T01:08:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451500+0000", 2024-04-26T01:08:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9342000000000001e-05 2024-04-26T01:08:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.456104+0000", 2024-04-26T01:08:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0046045119999999998 2024-04-26T01:08:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:121 3.0s0 3:0d858d01:::smithi16428812-41:head [append 0~688128,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.446304+0000", 2024-04-26T01:08:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 16.353720574, 2024-04-26T01:08:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0088967409999999997, 2024-04-26T01:08:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 121 2024-04-26T01:08:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446304+0000", 2024-04-26T01:08:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446304+0000", 2024-04-26T01:08:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446305+0000", 2024-04-26T01:08:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2529999999999999e-06 2024-04-26T01:08:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446941+0000", 2024-04-26T01:08:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00063629799999999999 2024-04-26T01:08:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446953+0000", 2024-04-26T01:08:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1752e-05 2024-04-26T01:08:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446959+0000", 2024-04-26T01:08:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.6620000000000002e-06 2024-04-26T01:08:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446999+0000", 2024-04-26T01:08:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.9696000000000001e-05 2024-04-26T01:08:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.447103+0000", 2024-04-26T01:08:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010429000000000001 2024-04-26T01:08:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.447803+0000", 2024-04-26T01:08:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00070063500000000002 2024-04-26T01:08:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.449711+0000", 2024-04-26T01:08:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001907569 2024-04-26T01:08:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455152+0000", 2024-04-26T01:08:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0054406630000000001 2024-04-26T01:08:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455201+0000", 2024-04-26T01:08:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.8922999999999999e-05 2024-04-26T01:08:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:123 3.0s0 3:0d858d01:::smithi16428812-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.446995+0000", 2024-04-26T01:08:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 16.353029571, 2024-04-26T01:08:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.01100367, 2024-04-26T01:08:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 123 2024-04-26T01:08:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446995+0000", 2024-04-26T01:08:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446995+0000", 2024-04-26T01:08:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446995+0000", 2024-04-26T01:08:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4400000000000001e-07 2024-04-26T01:08:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.446997+0000", 2024-04-26T01:08:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4160000000000001e-06 2024-04-26T01:08:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.447001+0000", 2024-04-26T01:08:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.4329999999999997e-06 2024-04-26T01:08:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.447005+0000", 2024-04-26T01:08:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.805e-06 2024-04-26T01:08:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.449827+0000", 2024-04-26T01:08:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0028225519999999999 2024-04-26T01:08:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.449912+0000", 2024-04-26T01:08:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.4765000000000006e-05 2024-04-26T01:08:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455664+0000", 2024-04-26T01:08:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0057516859999999998 2024-04-26T01:08:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.455725+0000", 2024-04-26T01:08:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0676e-05 2024-04-26T01:08:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.457999+0000", 2024-04-26T01:08:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0022739930000000002 2024-04-26T01:08:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:152 3.0s0 3:0230bbd3:::smithi16428812-50:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.489313+0000", 2024-04-26T01:08:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 16.310711822999998, 2024-04-26T01:08:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.015629423, 2024-04-26T01:08:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 152 2024-04-26T01:08:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489313+0000", 2024-04-26T01:08:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489313+0000", 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489314+0000", 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2399999999999996e-07 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489318+0000", 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3800000000000004e-06 2024-04-26T01:08:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489326+0000", 2024-04-26T01:08:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5459999999999992e-06 2024-04-26T01:08:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489335+0000", 2024-04-26T01:08:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9050000000000003e-06 2024-04-26T01:08:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.491271+0000", 2024-04-26T01:08:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0019354579999999999 2024-04-26T01:08:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.491416+0000", 2024-04-26T01:08:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000145396 2024-04-26T01:08:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492139+0000", 2024-04-26T01:08:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00072277400000000003 2024-04-26T01:08:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492918+0000", 2024-04-26T01:08:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00077903499999999997 2024-04-26T01:08:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504860+0000", 2024-04-26T01:08:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011941712 2024-04-26T01:08:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504942+0000", 2024-04-26T01:08:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2293000000000006e-05 2024-04-26T01:08:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:153 3.0s0 3:0230bbd3:::smithi16428812-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:08:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 16.310678158000002, 2024-04-26T01:08:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017766607, 2024-04-26T01:08:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 153 2024-04-26T01:08:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:08:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:08:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489347+0000", 2024-04-26T01:08:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9999999999999997e-07 2024-04-26T01:08:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489351+0000", 2024-04-26T01:08:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7309999999999999e-06 2024-04-26T01:08:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489359+0000", 2024-04-26T01:08:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.6619999999999994e-06 2024-04-26T01:08:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489366+0000", 2024-04-26T01:08:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.798e-06 2024-04-26T01:08:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492454+0000", 2024-04-26T01:08:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0030882399999999999 2024-04-26T01:08:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492562+0000", 2024-04-26T01:08:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107948 2024-04-26T01:08:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504997+0000", 2024-04-26T01:08:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012434176 2024-04-26T01:08:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.505136+0000", 2024-04-26T01:08:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013994300000000001 2024-04-26T01:08:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.507113+0000", 2024-04-26T01:08:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0019764090000000002 2024-04-26T01:08:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 33/30 ECSubWriteReply(tid=13, last_complete=33'9, committed=1, applied=1))", 2024-04-26T01:08:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.493174+0000", 2024-04-26T01:08:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 16.306850540999999, 2024-04-26T01:08:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011776930999999999, 2024-04-26T01:08:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.493174+0000", 2024-04-26T01:08:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.493174+0000", 2024-04-26T01:08:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.493174+0000", 2024-04-26T01:08:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9699999999999996e-07 2024-04-26T01:08:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.493177+0000", 2024-04-26T01:08:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0070000000000002e-06 2024-04-26T01:08:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504751+0000", 2024-04-26T01:08:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011573498999999999 2024-04-26T01:08:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504758+0000", 2024-04-26T01:08:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7159999999999997e-06 2024-04-26T01:08:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504810+0000", 2024-04-26T01:08:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2166000000000002e-05 2024-04-26T01:08:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504951+0000", 2024-04-26T01:08:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000141046 2024-04-26T01:08:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:284 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~753664] snapc 4=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:04.530659+0000", 2024-04-26T01:08:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.26936501, 2024-04-26T01:08:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.015592622, 2024-04-26T01:08:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 284 2024-04-26T01:08:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.530659+0000", 2024-04-26T01:08:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.530659+0000", 2024-04-26T01:08:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.530663+0000", 2024-04-26T01:08:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7529999999999999e-06 2024-04-26T01:08:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.531697+0000", 2024-04-26T01:08:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0010340410000000001 2024-04-26T01:08:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.531730+0000", 2024-04-26T01:08:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.3194000000000001e-05 2024-04-26T01:08:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.531756+0000", 2024-04-26T01:08:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5559999999999999e-05 2024-04-26T01:08:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.531934+0000", 2024-04-26T01:08:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017789499999999999 2024-04-26T01:08:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.532110+0000", 2024-04-26T01:08:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017660500000000001 2024-04-26T01:08:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.534200+0000", 2024-04-26T01:08:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0020893299999999999 2024-04-26T01:08:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.541721+0000", 2024-04-26T01:08:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0075214430000000001 2024-04-26T01:08:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.546130+0000", 2024-04-26T01:08:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0044085820000000003 2024-04-26T01:08:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.546252+0000", 2024-04-26T01:08:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012221899999999999 2024-04-26T01:08:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:285 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~376832] snapc 4=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:04.533234+0000", 2024-04-26T01:08:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.266790910999999, 2024-04-26T01:08:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.013317677, 2024-04-26T01:08:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 285 2024-04-26T01:08:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533234+0000", 2024-04-26T01:08:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533234+0000", 2024-04-26T01:08:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533237+0000", 2024-04-26T01:08:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.2030000000000001e-06 2024-04-26T01:08:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533827+0000", 2024-04-26T01:08:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00058986900000000003 2024-04-26T01:08:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533847+0000", 2024-04-26T01:08:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0789999999999999e-05 2024-04-26T01:08:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533859+0000", 2024-04-26T01:08:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.188e-05 2024-04-26T01:08:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.537616+0000", 2024-04-26T01:08:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0037569159999999999 2024-04-26T01:08:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.537883+0000", 2024-04-26T01:08:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00026640499999999999 2024-04-26T01:08:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.539825+0000", 2024-04-26T01:08:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001942781 2024-04-26T01:08:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.542144+0000", 2024-04-26T01:08:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0023188850000000001 2024-04-26T01:08:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.546450+0000", 2024-04-26T01:08:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0043053019999999996 2024-04-26T01:08:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.546551+0000", 2024-04-26T01:08:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000101646 2024-04-26T01:08:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:286 3.0s0 3:0982979f:::smithi16428812-35:head [setxattr _header (56) in=7b] snapc 4=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:04.533890+0000", 2024-04-26T01:08:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.266134779, 2024-04-26T01:08:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014544107000000001, 2024-04-26T01:08:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 286 2024-04-26T01:08:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533890+0000", 2024-04-26T01:08:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533890+0000", 2024-04-26T01:08:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533891+0000", 2024-04-26T01:08:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.686e-06 2024-04-26T01:08:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533914+0000", 2024-04-26T01:08:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3087000000000001e-05 2024-04-26T01:08:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533934+0000", 2024-04-26T01:08:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9522999999999999e-05 2024-04-26T01:08:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533949+0000", 2024-04-26T01:08:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4705999999999999e-05 2024-04-26T01:08:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.542306+0000", 2024-04-26T01:08:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0083571340000000004 2024-04-26T01:08:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.542483+0000", 2024-04-26T01:08:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000177405 2024-04-26T01:08:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.543919+0000", 2024-04-26T01:08:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0014354350000000001 2024-04-26T01:08:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.545371+0000", 2024-04-26T01:08:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001452152 2024-04-26T01:08:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.548178+0000", 2024-04-26T01:08:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0028074380000000002 2024-04-26T01:08:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.548434+0000", 2024-04-26T01:08:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00025554100000000003 2024-04-26T01:08:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:287 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:04.533969+0000", 2024-04-26T01:08:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.266055087, 2024-04-26T01:08:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.018210147999999999, 2024-04-26T01:08:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 287 2024-04-26T01:08:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533969+0000", 2024-04-26T01:08:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533969+0000", 2024-04-26T01:08:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533971+0000", 2024-04-26T01:08:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3629999999999999e-06 2024-04-26T01:08:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533977+0000", 2024-04-26T01:08:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5799999999999997e-06 2024-04-26T01:08:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.534001+0000", 2024-04-26T01:08:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4006999999999999e-05 2024-04-26T01:08:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.534016+0000", 2024-04-26T01:08:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5160000000000001e-05 2024-04-26T01:08:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.545071+0000", 2024-04-26T01:08:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011054938 2024-04-26T01:08:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.545349+0000", 2024-04-26T01:08:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00027734100000000001 2024-04-26T01:08:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.549341+0000", 2024-04-26T01:08:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0039925150000000003 2024-04-26T01:08:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.549639+0000", 2024-04-26T01:08:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00029756800000000001 2024-04-26T01:08:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.552180+0000", 2024-04-26T01:08:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.002540676 2024-04-26T01:08:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:367 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~589824] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.432060+0000", 2024-04-26T01:08:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 6.3679641240000002, 2024-04-26T01:08:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010466259, 2024-04-26T01:08:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 367 2024-04-26T01:08:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.432060+0000", 2024-04-26T01:08:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.432060+0000", 2024-04-26T01:08:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.432063+0000", 2024-04-26T01:08:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9660000000000001e-06 2024-04-26T01:08:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.433211+0000", 2024-04-26T01:08:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001147234 2024-04-26T01:08:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.433263+0000", 2024-04-26T01:08:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2389999999999998e-05 2024-04-26T01:08:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.433277+0000", 2024-04-26T01:08:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4256e-05 2024-04-26T01:08:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.433390+0000", 2024-04-26T01:08:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00011244299999999999 2024-04-26T01:08:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.433531+0000", 2024-04-26T01:08:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014179199999999999 2024-04-26T01:08:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.435144+0000", 2024-04-26T01:08:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0016122770000000001 2024-04-26T01:08:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.437591+0000", 2024-04-26T01:08:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0024474739999999998 2024-04-26T01:08:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.442355+0000", 2024-04-26T01:08:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0047643049999999999 2024-04-26T01:08:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.442527+0000", 2024-04-26T01:08:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017112200000000001 2024-04-26T01:08:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:368 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~573440] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.436812+0000", 2024-04-26T01:08:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 6.363212882, 2024-04-26T01:08:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011097516999999999, 2024-04-26T01:08:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 368 2024-04-26T01:08:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.436812+0000", 2024-04-26T01:08:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.436812+0000", 2024-04-26T01:08:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.436814+0000", 2024-04-26T01:08:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8930000000000001e-06 2024-04-26T01:08:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438177+0000", 2024-04-26T01:08:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001362156 2024-04-26T01:08:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438212+0000", 2024-04-26T01:08:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5233e-05 2024-04-26T01:08:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438235+0000", 2024-04-26T01:08:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2932999999999999e-05 2024-04-26T01:08:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438360+0000", 2024-04-26T01:08:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012524899999999999 2024-04-26T01:08:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438648+0000", 2024-04-26T01:08:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00028843100000000002 2024-04-26T01:08:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.440233+0000", 2024-04-26T01:08:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001584741 2024-04-26T01:08:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.447494+0000", 2024-04-26T01:08:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0072610319999999997 2024-04-26T01:08:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.447661+0000", 2024-04-26T01:08:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016724099999999999 2024-04-26T01:08:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.447909+0000", 2024-04-26T01:08:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00024760800000000002 2024-04-26T01:08:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:369 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~196608] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.438279+0000", 2024-04-26T01:08:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 6.3617451740000002, 2024-04-26T01:08:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010752640000000001, 2024-04-26T01:08:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 369 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438279+0000", 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438279+0000", 2024-04-26T01:08:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438281+0000", 2024-04-26T01:08:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1600000000000001e-06 2024-04-26T01:08:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438431+0000", 2024-04-26T01:08:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000149469 2024-04-26T01:08:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438450+0000", 2024-04-26T01:08:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8666000000000001e-05 2024-04-26T01:08:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438460+0000", 2024-04-26T01:08:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.045e-05 2024-04-26T01:08:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.442767+0000", 2024-04-26T01:08:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0043069550000000003 2024-04-26T01:08:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.442939+0000", 2024-04-26T01:08:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017222299999999999 2024-04-26T01:08:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.443947+0000", 2024-04-26T01:08:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0010079780000000001 2024-04-26T01:08:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.447923+0000", 2024-04-26T01:08:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0039753280000000002 2024-04-26T01:08:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.448892+0000", 2024-04-26T01:08:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00096973199999999995 2024-04-26T01:08:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.449032+0000", 2024-04-26T01:08:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000139679 2024-04-26T01:08:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:370 3.0s0 3:0982979f:::smithi16428812-35:head [setxattr _header (56) in=7b] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.438488+0000", 2024-04-26T01:08:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 6.3615359590000002, 2024-04-26T01:08:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011425454999999999, 2024-04-26T01:08:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 370 2024-04-26T01:08:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438488+0000", 2024-04-26T01:08:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438488+0000", 2024-04-26T01:08:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438490+0000", 2024-04-26T01:08:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.55e-06 2024-04-26T01:08:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438498+0000", 2024-04-26T01:08:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5299999999999999e-06 2024-04-26T01:08:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438515+0000", 2024-04-26T01:08:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7873e-05 2024-04-26T01:08:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438532+0000", 2024-04-26T01:08:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6932999999999999e-05 2024-04-26T01:08:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.445253+0000", 2024-04-26T01:08:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0067210630000000002 2024-04-26T01:08:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.445498+0000", 2024-04-26T01:08:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000244095 2024-04-26T01:08:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.446777+0000", 2024-04-26T01:08:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001279046 2024-04-26T01:08:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.448516+0000", 2024-04-26T01:08:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0017391730000000001 2024-04-26T01:08:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.449787+0000", 2024-04-26T01:08:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001271503 2024-04-26T01:08:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.449914+0000", 2024-04-26T01:08:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012668900000000001 2024-04-26T01:08:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:371 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.438556+0000", 2024-04-26T01:08:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 6.3614685160000004, 2024-04-26T01:08:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.015156252, 2024-04-26T01:08:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 371 2024-04-26T01:08:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438556+0000", 2024-04-26T01:08:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438556+0000", 2024-04-26T01:08:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438557+0000", 2024-04-26T01:08:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4699999999999999e-06 2024-04-26T01:08:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438563+0000", 2024-04-26T01:08:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7699999999999998e-06 2024-04-26T01:08:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438608+0000", 2024-04-26T01:08:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.4898999999999998e-05 2024-04-26T01:08:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438623+0000", 2024-04-26T01:08:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.47e-05 2024-04-26T01:08:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.450782+0000", 2024-04-26T01:08:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012158983 2024-04-26T01:08:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.450982+0000", 2024-04-26T01:08:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000200602 2024-04-26T01:08:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.453712+0000", 2024-04-26T01:08:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0027298280000000001 2024-04-26T01:08:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:15.909 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:15.910 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:08:15.911 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:08:15.912 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:15.912+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:16.012 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:16.008+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:16.036 INFO:tasks.rados.rados.0.smithi164.stdout:190: done (0 left) 2024-04-26T01:08:16.036 INFO:tasks.rados.rados.0.smithi164.stdout:191: read oid 47 snap -1 2024-04-26T01:08:16.037 INFO:tasks.rados.rados.0.smithi164.stdout:191: expect (ObjNum 65 snap 3 seq_num 65) 2024-04-26T01:08:16.037 INFO:tasks.rados.rados.0.smithi164.stdout:192: setattr oid 22 current snap is 10 2024-04-26T01:08:16.044 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:16.044 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:16.044 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:08:16.044 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:16.046 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 49 current snap is 10 2024-04-26T01:08:16.046 INFO:tasks.rados.rados.0.smithi164.stdout:193: seq_num 97 ranges {819200=475136,1294336=442368,1736704=442368,2179072=278528} 2024-04-26T01:08:16.062 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:08:16.068 INFO:tasks.rados.rados.0.smithi164.stdout:193: writing smithi16428812-49 from 819200 to 1294336 tid 1 2024-04-26T01:08:16.076 INFO:tasks.rados.rados.0.smithi164.stdout:193: writing smithi16428812-49 from 1294336 to 1736704 tid 2 2024-04-26T01:08:16.084 INFO:tasks.rados.rados.0.smithi164.stdout:193: writing smithi16428812-49 from 1736704 to 2179072 tid 3 2024-04-26T01:08:16.089 INFO:tasks.rados.rados.0.smithi164.stdout:193: writing smithi16428812-49 from 2179072 to 2457600 tid 4 2024-04-26T01:08:16.089 INFO:tasks.rados.rados.0.smithi164.stdout:194: snap_remove snap 8 2024-04-26T01:08:16.108 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:16.108 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:16.113 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:16.109+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:16.133 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:16.134 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:16.158 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:16.158 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:16.186 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:16.186 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:16.198 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:16.198 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:16.213 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:16.212+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:16.214 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:16.215 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:16.216 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:08:16.246 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:16.247 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:16.279 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:16.280 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:16.312 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:16.312 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:16.314 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:16.312+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:16.341 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:16.342 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:16.374 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:16.374 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:16.381 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:16.381 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:16.396 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:08:16.406 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:16.406 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:16.406 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-26T01:08:16.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:16.412+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:16.438 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:16.438 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:16.474 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:16.474 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:16.509 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:16.509 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:16.512 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:16.512 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:16.515 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:16.512+0000 7f71dcfad640 0 -- 172.21.15.8:0/3591563832 >> [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] conn(0x7f71d8223f60 msgr2=0x7f71d8226350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:08:16.516 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:16.512+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:16.526 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:08:16.552 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:16.552 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:16.558 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T01:08:16.558 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T01:08:16.617 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:16.612+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:16.648 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:16.648 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:16.662 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:08:16.718 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:16.716+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:16.781 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:16.781 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:16.795 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:08:16.819 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:16.816+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:16.920 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:16.920+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:16.921 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:16.921 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:16.938 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:08:16.940 INFO:teuthology.orchestra.run.smithi164.stdout:get_pg_num_history pg_num_history pg_num_history(e42 pg_nums {1={23=1},2={25=8},3={30=16}} deleted_pools ) 2024-04-26T01:08:16.940 INFO:teuthology.orchestra.run.smithi164.stdout:Importing pgid 3.as6 2024-04-26T01:08:16.940 INFO:teuthology.orchestra.run.smithi164.stdout:Write 6#3:592ac288:::smithi16428812-20:head# 2024-04-26T01:08:16.940 INFO:teuthology.orchestra.run.smithi164.stdout:snapset 5=[]:{} 2024-04-26T01:08:16.942 INFO:teuthology.orchestra.run.smithi164.stdout:write_pg epoch 41 info 3.as6( v 41'10 (0'0,41'10] local-lis/les=30/31 n=1 ec=30/30 lis/c=30/30 les/c/f=31/31/0 sis=30) 2024-04-26T01:08:16.942 INFO:teuthology.orchestra.run.smithi164.stdout:Import successful 2024-04-26T01:08:17.021 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:17.020+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:17.053 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:17.053 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:17.068 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:08:17.068 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-26T01:08:17.098 INFO:tasks.rados.rados.0.smithi164.stdout:194: done (3 left) 2024-04-26T01:08:17.098 INFO:tasks.rados.rados.0.smithi164.stdout:195: copy_from oid 50 from oid 18 current snap is 10 2024-04-26T01:08:17.098 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 22 v 32 (ObjNum 95 snap 8 seq_num 95) dirty exists 2024-04-26T01:08:17.099 INFO:tasks.rados.rados.0.smithi164.stdout:193: finishing write tid 1 to smithi16428812-49 2024-04-26T01:08:17.099 INFO:tasks.rados.rados.0.smithi164.stdout:191: expect (ObjNum 65 snap 3 seq_num 65) 2024-04-26T01:08:17.123 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:17.117+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:17.194 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:17.195 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:17.208 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:08:17.215 INFO:tasks.rados.rados.0.smithi164.stdout:193: finishing write tid 2 to smithi16428812-49 2024-04-26T01:08:17.215 INFO:tasks.rados.rados.0.smithi164.stdout:193: finishing write tid 3 to smithi16428812-49 2024-04-26T01:08:17.215 INFO:tasks.rados.rados.0.smithi164.stdout:193: finishing write tid 4 to smithi16428812-49 2024-04-26T01:08:17.215 INFO:tasks.rados.rados.0.smithi164.stdout:193: finishing write tid 5 to smithi16428812-49 2024-04-26T01:08:17.215 INFO:tasks.rados.rados.0.smithi164.stdout:193: finishing write tid 6 to smithi16428812-49 2024-04-26T01:08:17.215 INFO:tasks.rados.rados.0.smithi164.stdout:193: oid 49 updating version 0 to 42 2024-04-26T01:08:17.215 INFO:tasks.rados.rados.0.smithi164.stdout:193: oid 49 version 42 is already newer than 41 2024-04-26T01:08:17.215 INFO:tasks.rados.rados.0.smithi164.stdout:193: oid 49 version 42 is already newer than 40 2024-04-26T01:08:17.216 INFO:tasks.rados.rados.0.smithi164.stdout:193: oid 49 version 42 is already newer than 39 2024-04-26T01:08:17.216 INFO:tasks.rados.rados.0.smithi164.stdout:193: oid 49 version 42 is already newer than 38 2024-04-26T01:08:17.216 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 49 v 42 (ObjNum 97 snap 10 seq_num 97) dirty exists 2024-04-26T01:08:17.216 INFO:tasks.rados.rados.0.smithi164.stdout:193: left oid 49 (ObjNum 97 snap 10 seq_num 97) 2024-04-26T01:08:17.216 INFO:tasks.rados.rados.0.smithi164.stdout:195: finishing copy_from racing read to smithi16428812-50 2024-04-26T01:08:17.216 INFO:tasks.rados.rados.0.smithi164.stdout:195: finishing copy_from to smithi16428812-50 2024-04-26T01:08:17.216 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 50 v 30 (ObjNum 84 snap 6 seq_num 84) dirty exists 2024-04-26T01:08:17.216 INFO:tasks.rados.rados.0.smithi164.stdout:191: done (3 left) 2024-04-26T01:08:17.216 INFO:tasks.rados.rados.0.smithi164.stdout:192: done (2 left) 2024-04-26T01:08:17.216 INFO:tasks.rados.rados.0.smithi164.stdout:193: done (1 left) 2024-04-26T01:08:17.216 INFO:tasks.rados.rados.0.smithi164.stdout:195: done (0 left) 2024-04-26T01:08:17.216 INFO:tasks.rados.rados.0.smithi164.stdout:196: read oid 2 snap -1 2024-04-26T01:08:17.216 INFO:tasks.rados.rados.0.smithi164.stdout:196: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-26T01:08:17.216 INFO:tasks.rados.rados.0.smithi164.stdout:197: read oid 20 snap 9 2024-04-26T01:08:17.217 INFO:tasks.rados.rados.0.smithi164.stdout:197: expect (ObjNum 69 snap 4 seq_num 69) 2024-04-26T01:08:17.217 INFO:tasks.rados.rados.0.smithi164.stdout:198: snap_create 2024-04-26T01:08:17.221 INFO:tasks.rados.rados.0.smithi164.stdout:196: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-26T01:08:17.226 INFO:tasks.rados.rados.0.smithi164.stdout:197: expect (ObjNum 69 snap 4 seq_num 69) 2024-04-26T01:08:17.254 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:17.252+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:17.325 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:17.321+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:17.329 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:17.329 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:17.344 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:08:17.426 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:17.421+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:17.480 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f /var/log/ceph/exp.3.as6.11 2024-04-26T01:08:17.501 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:17.501 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:17.515 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:08:17.527 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:17.521+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:17.535 INFO:tasks.thrashosds.thrasher:checking if target objectstore is bluestore on osd.11 2024-04-26T01:08:17.535 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-11 show-label 2024-04-26T01:08:17.617 INFO:teuthology.orchestra.run.smithi164.stdout:inferring bluefs devices from bluestore path 2024-04-26T01:08:17.617 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:17.617 INFO:teuthology.orchestra.run.smithi164.stdout: "/var/lib/ceph/osd/ceph-11/block": { 2024-04-26T01:08:17.617 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_uuid": "9bfdf353-5174-40dd-bb3c-ab2dc850adad", 2024-04-26T01:08:17.617 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 96636764160, 2024-04-26T01:08:17.617 INFO:teuthology.orchestra.run.smithi164.stdout: "btime": "2024-04-26T01:07:16.873414+0000", 2024-04-26T01:08:17.618 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "main", 2024-04-26T01:08:17.618 INFO:teuthology.orchestra.run.smithi164.stdout: "bfm_blocks": "23592960", 2024-04-26T01:08:17.618 INFO:teuthology.orchestra.run.smithi164.stdout: "bfm_blocks_per_key": "128", 2024-04-26T01:08:17.618 INFO:teuthology.orchestra.run.smithi164.stdout: "bfm_bytes_per_block": "4096", 2024-04-26T01:08:17.618 INFO:teuthology.orchestra.run.smithi164.stdout: "bfm_size": "96636764160", 2024-04-26T01:08:17.618 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs": "1", 2024-04-26T01:08:17.618 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_fsid": "4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554", 2024-04-26T01:08:17.618 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_version_when_created": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-26T01:08:17.618 INFO:teuthology.orchestra.run.smithi164.stdout: "created_at": "2024-04-26T01:07:19.301268Z", 2024-04-26T01:08:17.618 INFO:teuthology.orchestra.run.smithi164.stdout: "elastic_shared_blobs": "1", 2024-04-26T01:08:17.618 INFO:teuthology.orchestra.run.smithi164.stdout: "kv_backend": "rocksdb", 2024-04-26T01:08:17.618 INFO:teuthology.orchestra.run.smithi164.stdout: "magic": "ceph osd volume v026", 2024-04-26T01:08:17.618 INFO:teuthology.orchestra.run.smithi164.stdout: "mkfs_done": "yes", 2024-04-26T01:08:17.619 INFO:teuthology.orchestra.run.smithi164.stdout: "ready": "ready", 2024-04-26T01:08:17.619 INFO:teuthology.orchestra.run.smithi164.stdout: "require_osd_release": "19", 2024-04-26T01:08:17.619 INFO:teuthology.orchestra.run.smithi164.stdout: "whoami": "11" 2024-04-26T01:08:17.619 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:17.619 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:17.620 INFO:tasks.thrashosds.thrasher:checking if target bluestore supports sharding on osd.11 2024-04-26T01:08:17.620 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-11 show-sharding 2024-04-26T01:08:17.628 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:17.628+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:17.640 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:17.640 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:17.655 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:08:17.729 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:17.729+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:17.783 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:17.783 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:17.797 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:08:17.842 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:17.840+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:17.919 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:17.919 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:17.930 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:17.928+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:17.933 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:08:18.032 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:18.028+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:18.100 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-04-26T01:08:18.102 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:18.102 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:18.106 INFO:tasks.rados.rados.0.smithi164.stdout:196: done (2 left) 2024-04-26T01:08:18.106 INFO:tasks.rados.rados.0.smithi164.stdout:197: done (1 left) 2024-04-26T01:08:18.106 INFO:tasks.rados.rados.0.smithi164.stdout:198: done (0 left) 2024-04-26T01:08:18.106 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 29 current snap is 11 2024-04-26T01:08:18.106 INFO:tasks.rados.rados.0.smithi164.stdout:199: seq_num 98 ranges {475136=491520,966656=589824,1556480=557056,2113536=16384} 2024-04-26T01:08:18.119 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:08:18.122 INFO:tasks.rados.rados.0.smithi164.stdout:199: writing smithi16428812-29 from 475136 to 966656 tid 1 2024-04-26T01:08:18.132 INFO:tasks.rados.rados.0.smithi164.stdout:199: writing smithi16428812-29 from 966656 to 1556480 tid 2 2024-04-26T01:08:18.141 INFO:tasks.rados.rados.0.smithi164.stdout:199: writing smithi16428812-29 from 1556480 to 2113536 tid 3 2024-04-26T01:08:18.141 INFO:tasks.rados.rados.0.smithi164.stdout:199: writing smithi16428812-29 from 2113536 to 2129920 tid 4 2024-04-26T01:08:18.142 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 9 current snap is 11 2024-04-26T01:08:18.142 INFO:tasks.rados.rados.0.smithi164.stdout:199: finishing write tid 1 to smithi16428812-29 2024-04-26T01:08:18.142 INFO:tasks.rados.rados.0.smithi164.stdout:199: finishing write tid 2 to smithi16428812-29 2024-04-26T01:08:18.142 INFO:tasks.rados.rados.0.smithi164.stdout:200: seq_num 99 ranges {0=507904,507904=606208,1114112=557056} 2024-04-26T01:08:18.150 INFO:tasks.rados.rados.0.smithi164.stdout:200: writing smithi16428812-9 from 0 to 507904 tid 1 2024-04-26T01:08:18.160 INFO:tasks.rados.rados.0.smithi164.stdout:200: writing smithi16428812-9 from 507904 to 1114112 tid 2 2024-04-26T01:08:18.167 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:18.164+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:18.169 INFO:tasks.rados.rados.0.smithi164.stdout:200: writing smithi16428812-9 from 1114112 to 1671168 tid 3 2024-04-26T01:08:18.169 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 7 current snap is 11 2024-04-26T01:08:18.169 INFO:tasks.rados.rados.0.smithi164.stdout:201: seq_num 100 ranges {1556480=524288,2080768=753664,2834432=409600} 2024-04-26T01:08:18.193 INFO:tasks.rados.rados.0.smithi164.stdout:201: writing smithi16428812-7 from 1556480 to 2080768 tid 1 2024-04-26T01:08:18.200 INFO:tasks.rados.rados.0.smithi164.stdout:201: writing smithi16428812-7 from 2080768 to 2834432 tid 2 2024-04-26T01:08:18.204 INFO:tasks.rados.rados.0.smithi164.stdout:201: writing smithi16428812-7 from 2834432 to 3244032 tid 3 2024-04-26T01:08:18.204 INFO:tasks.rados.rados.0.smithi164.stdout:202: delete oid 46 current snap is 11 2024-04-26T01:08:18.204 INFO:tasks.rados.rados.0.smithi164.stdout:199: finishing write tid 3 to smithi16428812-29 2024-04-26T01:08:18.205 INFO:tasks.rados.rados.0.smithi164.stdout:199: finishing write tid 4 to smithi16428812-29 2024-04-26T01:08:18.205 INFO:tasks.rados.rados.0.smithi164.stdout:199: finishing write tid 5 to smithi16428812-29 2024-04-26T01:08:18.205 INFO:tasks.rados.rados.0.smithi164.stdout:199: finishing write tid 6 to smithi16428812-29 2024-04-26T01:08:18.205 INFO:tasks.rados.rados.0.smithi164.stdout:199: oid 29 updating version 0 to 21 2024-04-26T01:08:18.205 INFO:tasks.rados.rados.0.smithi164.stdout:199: oid 29 updating version 21 to 23 2024-04-26T01:08:18.205 INFO:tasks.rados.rados.0.smithi164.stdout:199: oid 29 version 23 is already newer than 22 2024-04-26T01:08:18.205 INFO:tasks.rados.rados.0.smithi164.stdout:199: oid 29 version 23 is already newer than 20 2024-04-26T01:08:18.205 INFO:tasks.rados.rados.0.smithi164.stdout:199: oid 29 updating version 23 to 24 2024-04-26T01:08:18.205 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 29 v 24 (ObjNum 98 snap 11 seq_num 98) dirty exists 2024-04-26T01:08:18.205 INFO:tasks.rados.rados.0.smithi164.stdout:199: left oid 29 (ObjNum 98 snap 11 seq_num 98) 2024-04-26T01:08:18.205 INFO:tasks.rados.rados.0.smithi164.stdout:200: finishing write tid 1 to smithi16428812-9 2024-04-26T01:08:18.205 INFO:tasks.rados.rados.0.smithi164.stdout:200: finishing write tid 2 to smithi16428812-9 2024-04-26T01:08:18.205 INFO:tasks.rados.rados.0.smithi164.stdout:200: finishing write tid 3 to smithi16428812-9 2024-04-26T01:08:18.205 INFO:tasks.rados.rados.0.smithi164.stdout:201: finishing write tid 1 to smithi16428812-7 2024-04-26T01:08:18.209 INFO:tasks.rados.rados.0.smithi164.stdout:201: finishing write tid 2 to smithi16428812-7 2024-04-26T01:08:18.209 INFO:tasks.rados.rados.0.smithi164.stdout:201: finishing write tid 3 to smithi16428812-7 2024-04-26T01:08:18.210 INFO:tasks.rados.rados.0.smithi164.stdout:201: finishing write tid 4 to smithi16428812-7 2024-04-26T01:08:18.211 INFO:tasks.rados.rados.0.smithi164.stdout:199: done (3 left) 2024-04-26T01:08:18.211 INFO:tasks.rados.rados.0.smithi164.stdout:202: done (2 left) 2024-04-26T01:08:18.211 INFO:tasks.rados.rados.0.smithi164.stdout:203: read oid 2 snap -1 2024-04-26T01:08:18.211 INFO:tasks.rados.rados.0.smithi164.stdout:203: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-26T01:08:18.212 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 24 current snap is 11 2024-04-26T01:08:18.212 INFO:tasks.rados.rados.0.smithi164.stdout:204: seq_num 102 ranges {393216=557056,950272=638976,1589248=524288} 2024-04-26T01:08:18.219 INFO:tasks.rados.rados.0.smithi164.stdout:204: writing smithi16428812-24 from 393216 to 950272 tid 1 2024-04-26T01:08:18.225 INFO:tasks.rados.rados.0.smithi164.stdout:204: writing smithi16428812-24 from 950272 to 1589248 tid 2 2024-04-26T01:08:18.231 INFO:tasks.rados.rados.0.smithi164.stdout:204: writing smithi16428812-24 from 1589248 to 2113536 tid 3 2024-04-26T01:08:18.231 INFO:tasks.rados.rados.0.smithi164.stdout:205: delete oid 34 current snap is 11 2024-04-26T01:08:18.231 INFO:tasks.rados.rados.0.smithi164.stdout:201: finishing write tid 5 to smithi16428812-7 2024-04-26T01:08:18.231 INFO:tasks.rados.rados.0.smithi164.stdout:201: oid 7 updating version 0 to 40 2024-04-26T01:08:18.231 INFO:tasks.rados.rados.0.smithi164.stdout:201: oid 7 version 40 is already newer than 38 2024-04-26T01:08:18.231 INFO:tasks.rados.rados.0.smithi164.stdout:201: oid 7 updating version 40 to 41 2024-04-26T01:08:18.231 INFO:tasks.rados.rados.0.smithi164.stdout:201: oid 7 version 41 is already newer than 39 2024-04-26T01:08:18.231 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 7 v 41 (ObjNum 100 snap 11 seq_num 100) dirty exists 2024-04-26T01:08:18.231 INFO:tasks.rados.rados.0.smithi164.stdout:201: left oid 7 (ObjNum 100 snap 11 seq_num 100) 2024-04-26T01:08:18.231 INFO:tasks.rados.rados.0.smithi164.stdout:203: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-26T01:08:18.232 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:18.228+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:18.233 INFO:tasks.rados.rados.0.smithi164.stdout:204: finishing write tid 1 to smithi16428812-24 2024-04-26T01:08:18.236 INFO:tasks.rados.rados.0.smithi164.stdout:204: finishing write tid 2 to smithi16428812-24 2024-04-26T01:08:18.239 INFO:tasks.rados.rados.0.smithi164.stdout:204: finishing write tid 3 to smithi16428812-24 2024-04-26T01:08:18.240 INFO:tasks.rados.rados.0.smithi164.stdout:204: finishing write tid 4 to smithi16428812-24 2024-04-26T01:08:18.243 INFO:tasks.rados.rados.0.smithi164.stdout:204: finishing write tid 5 to smithi16428812-24 2024-04-26T01:08:18.243 INFO:tasks.rados.rados.0.smithi164.stdout:204: oid 24 updating version 0 to 26 2024-04-26T01:08:18.243 INFO:tasks.rados.rados.0.smithi164.stdout:204: oid 24 updating version 26 to 27 2024-04-26T01:08:18.243 INFO:tasks.rados.rados.0.smithi164.stdout:204: oid 24 updating version 27 to 28 2024-04-26T01:08:18.243 INFO:tasks.rados.rados.0.smithi164.stdout:204: oid 24 updating version 28 to 29 2024-04-26T01:08:18.243 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 24 v 29 (ObjNum 102 snap 11 seq_num 102) dirty exists 2024-04-26T01:08:18.243 INFO:tasks.rados.rados.0.smithi164.stdout:204: left oid 24 (ObjNum 102 snap 11 seq_num 102) 2024-04-26T01:08:18.294 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:18.294 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:18.309 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:08:18.312 INFO:tasks.rados.rados.0.smithi164.stdout:200: finishing write tid 4 to smithi16428812-9 2024-04-26T01:08:18.316 INFO:tasks.rados.rados.0.smithi164.stdout:200: finishing write tid 5 to smithi16428812-9 2024-04-26T01:08:18.316 INFO:tasks.rados.rados.0.smithi164.stdout:200: oid 9 updating version 0 to 39 2024-04-26T01:08:18.316 INFO:tasks.rados.rados.0.smithi164.stdout:200: oid 9 version 39 is already newer than 38 2024-04-26T01:08:18.316 INFO:tasks.rados.rados.0.smithi164.stdout:200: oid 9 updating version 39 to 41 2024-04-26T01:08:18.316 INFO:tasks.rados.rados.0.smithi164.stdout:200: oid 9 version 41 is already newer than 40 2024-04-26T01:08:18.316 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 9 v 41 (ObjNum 99 snap 11 seq_num 99) dirty exists 2024-04-26T01:08:18.316 INFO:tasks.rados.rados.0.smithi164.stdout:200: left oid 9 (ObjNum 99 snap 11 seq_num 99) 2024-04-26T01:08:18.334 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:18.336+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:18.435 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:18.432+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:18.444 INFO:tasks.rados.rados.0.smithi164.stdout:200: done (4 left) 2024-04-26T01:08:18.444 INFO:tasks.rados.rados.0.smithi164.stdout:201: done (3 left) 2024-04-26T01:08:18.444 INFO:tasks.rados.rados.0.smithi164.stdout:203: done (2 left) 2024-04-26T01:08:18.444 INFO:tasks.rados.rados.0.smithi164.stdout:204: done (1 left) 2024-04-26T01:08:18.444 INFO:tasks.rados.rados.0.smithi164.stdout:205: done (0 left) 2024-04-26T01:08:18.444 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 40 current snap is 11 2024-04-26T01:08:18.444 INFO:tasks.rados.rados.0.smithi164.stdout:206: seq_num 104 ranges {0=589824,589824=540672,1130496=622592} 2024-04-26T01:08:18.449 INFO:tasks.rados.rados.0.smithi164.stdout:206: writing smithi16428812-40 from 0 to 589824 tid 1 2024-04-26T01:08:18.454 INFO:tasks.rados.rados.0.smithi164.stdout:206: writing smithi16428812-40 from 589824 to 1130496 tid 2 2024-04-26T01:08:18.460 INFO:tasks.rados.rados.0.smithi164.stdout:206: writing smithi16428812-40 from 1130496 to 1753088 tid 3 2024-04-26T01:08:18.460 INFO:tasks.rados.rados.0.smithi164.stdout:207: snap_create 2024-04-26T01:08:18.460 INFO:tasks.rados.rados.0.smithi164.stdout:206: finishing write tid 1 to smithi16428812-40 2024-04-26T01:08:18.467 INFO:tasks.rados.rados.0.smithi164.stdout:206: finishing write tid 2 to smithi16428812-40 2024-04-26T01:08:18.468 INFO:tasks.rados.rados.0.smithi164.stdout:206: finishing write tid 3 to smithi16428812-40 2024-04-26T01:08:18.478 INFO:teuthology.orchestra.run.smithi164.stdout:m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 2024-04-26T01:08:18.480 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:18.480 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:18.485 INFO:tasks.thrashosds.thrasher:applying new sharding to bluestore on osd.11 2024-04-26T01:08:18.485 INFO:tasks.thrashosds.thrasher:generate random sharding 2024-04-26T01:08:18.485 INFO:tasks.thrashosds.thrasher:applying new sharding: O(3,8-24) 2024-04-26T01:08:18.485 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-11 --sharding 'O(3,8-24)' reshard 2024-04-26T01:08:18.494 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:08:18.538 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:18.536+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:18.561 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:08:18.618 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:18.618 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:18.634 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:08:18.637 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:18.636+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:18.711 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:18.712 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:18.737 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:18.737 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:18.738 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:18.733+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:18.752 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:18.752 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:18.765 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:18.765 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:18.767 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:08:18.794 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:18.795 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:18.821 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:18.838 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:18.840+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:18.850 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:18.850 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:18.881 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:18.881 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:18.883 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:18.883 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:18.898 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:08:18.909 INFO:teuthology.orchestra.run.smithi164.stdout:reshard success 2024-04-26T01:08:18.911 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:18.911 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:18.939 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:18.933+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:18.940 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:18.940 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:18.973 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:18.973 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:19.006 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:19.006 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:19.006 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-26T01:08:19.030 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:19.030 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:19.035 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:19.036 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:19.039 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:19.036+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:19.045 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:08:19.065 INFO:tasks.rados.rados.0.smithi164.stdout:206: finishing write tid 4 to smithi16428812-40 2024-04-26T01:08:19.067 INFO:tasks.rados.rados.0.smithi164.stdout:206: finishing write tid 5 to smithi16428812-40 2024-04-26T01:08:19.067 INFO:tasks.rados.rados.0.smithi164.stdout:206: oid 40 updating version 0 to 26 2024-04-26T01:08:19.067 INFO:tasks.rados.rados.0.smithi164.stdout:206: oid 40 version 26 is already newer than 25 2024-04-26T01:08:19.067 INFO:tasks.rados.rados.0.smithi164.stdout:206: oid 40 version 26 is already newer than 23 2024-04-26T01:08:19.067 INFO:tasks.rados.rados.0.smithi164.stdout:206: oid 40 version 26 is already newer than 24 2024-04-26T01:08:19.067 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 40 v 26 (ObjNum 104 snap 11 seq_num 104) dirty exists 2024-04-26T01:08:19.067 INFO:tasks.rados.rados.0.smithi164.stdout:206: left oid 40 (ObjNum 104 snap 11 seq_num 104) 2024-04-26T01:08:19.068 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:19.068 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:19.098 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:19.099 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:19.106 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:19.104+0000 7fed512e2640 0 -- 172.21.15.8:0/257525276 >> [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] conn(0x7fed4c221740 msgr2=0x7fed4c223b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:08:19.111 INFO:tasks.rados.rados.0.smithi164.stdout:206: done (1 left) 2024-04-26T01:08:19.111 INFO:tasks.rados.rados.0.smithi164.stdout:207: done (0 left) 2024-04-26T01:08:19.111 INFO:tasks.rados.rados.0.smithi164.stdout:208: copy_from oid 36 from oid 20 current snap is 12 2024-04-26T01:08:19.111 INFO:tasks.rados.rados.0.smithi164.stdout:209: snap_create 2024-04-26T01:08:19.135 INFO:tasks.rados.rados.0.smithi164.stdout:208: finishing copy_from racing read to smithi16428812-36 2024-04-26T01:08:19.140 INFO:tasks.rados.rados.0.smithi164.stdout:208: finishing copy_from to smithi16428812-36 2024-04-26T01:08:19.140 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 36 v 6 (ObjNum 69 snap 4 seq_num 69) dirty exists 2024-04-26T01:08:19.141 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:19.140+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:19.141 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:19.142 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:19.146 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T01:08:19.147 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T01:08:19.184 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:19.184 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: "num_ops": 0 2024-04-26T01:08:19.185 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:19.199 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:08:19.241 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:19.240+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:19.318 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:19.318 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:19.319 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:08:19.319 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:08:19.319 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:19.333 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:08:19.357 INFO:tasks.thrashosds.thrasher:running fsck to verify new sharding on osd.11 2024-04-26T01:08:19.357 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-11 fsck 2024-04-26T01:08:19.380 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:19.377+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:19.442 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:19.437+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:19.459 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:19.460 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:08:19.460 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:08:19.460 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:08:19.460 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.460 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.2s5 32/30 ECSubWrite(tid=157, reqid=client.4521.0:13, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:19.460 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.308314+0000", 2024-04-26T01:08:19.460 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.149470555000001, 2024-04-26T01:08:19.460 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011288362999999999, 2024-04-26T01:08:19.460 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.460 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.460 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.461 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.461 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.461 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.461 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308314+0000", 2024-04-26T01:08:19.461 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.461 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.461 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.461 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308314+0000", 2024-04-26T01:08:19.461 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.461 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.461 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.461 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308315+0000", 2024-04-26T01:08:19.462 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.6599999999999995e-07 2024-04-26T01:08:19.462 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.462 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.462 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.462 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308467+0000", 2024-04-26T01:08:19.462 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000152249 2024-04-26T01:08:19.462 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.462 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.462 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.462 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308602+0000", 2024-04-26T01:08:19.462 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000135048 2024-04-26T01:08:19.462 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.462 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.463 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.463 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308653+0000", 2024-04-26T01:08:19.463 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0864e-05 2024-04-26T01:08:19.463 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.463 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.463 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.463 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308834+0000", 2024-04-26T01:08:19.463 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000180466 2024-04-26T01:08:19.463 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.463 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.463 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.463 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308857+0000", 2024-04-26T01:08:19.463 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2904e-05 2024-04-26T01:08:19.464 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.464 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.464 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.464 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.319506+0000", 2024-04-26T01:08:19.464 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010649141000000001 2024-04-26T01:08:19.464 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.464 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.464 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.464 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.319603+0000", 2024-04-26T01:08:19.464 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6924999999999998e-05 2024-04-26T01:08:19.464 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.464 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.464 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.465 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 32/30 ECSubWrite(tid=0, reqid=client.4521.0:16, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:19.465 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:19.465 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.147252236, 2024-04-26T01:08:19.465 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012119314000000001, 2024-04-26T01:08:19.465 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.465 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.465 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.465 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.465 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.465 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:19.466 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.466 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.466 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.466 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:19.466 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.466 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.466 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.466 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:19.466 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.7000000000000004e-07 2024-04-26T01:08:19.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.466 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.467 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.467 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310733+0000", 2024-04-26T01:08:19.467 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019985599999999999 2024-04-26T01:08:19.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.467 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.467 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310792+0000", 2024-04-26T01:08:19.467 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.8894000000000002e-05 2024-04-26T01:08:19.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.467 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.467 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310816+0000", 2024-04-26T01:08:19.467 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4094e-05 2024-04-26T01:08:19.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.468 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.468 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311022+0000", 2024-04-26T01:08:19.468 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00020608599999999999 2024-04-26T01:08:19.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.468 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.468 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311044+0000", 2024-04-26T01:08:19.468 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1889999999999999e-05 2024-04-26T01:08:19.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.468 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.468 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322407+0000", 2024-04-26T01:08:19.469 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011362778 2024-04-26T01:08:19.469 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.469 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.469 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.469 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322652+0000", 2024-04-26T01:08:19.469 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00024504600000000001 2024-04-26T01:08:19.469 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.469 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.469 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.469 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.469 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.469 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.es2 33/30 ECSubWrite(tid=0, reqid=client.4521.0:22, at_version=33'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:19.469 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.311344+0000", 2024-04-26T01:08:19.469 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.146440722000001, 2024-04-26T01:08:19.470 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010979113, 2024-04-26T01:08:19.470 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.470 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.470 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.470 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.470 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.470 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.470 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311344+0000", 2024-04-26T01:08:19.470 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.470 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.470 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.470 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.470 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311344+0000", 2024-04-26T01:08:19.471 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.471 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.471 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.471 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.471 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311351+0000", 2024-04-26T01:08:19.471 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1040000000000001e-06 2024-04-26T01:08:19.471 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.471 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.471 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.471 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311496+0000", 2024-04-26T01:08:19.471 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000144852 2024-04-26T01:08:19.471 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.471 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.471 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.472 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311639+0000", 2024-04-26T01:08:19.472 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014267800000000001 2024-04-26T01:08:19.472 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.472 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.472 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.472 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311663+0000", 2024-04-26T01:08:19.472 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4090000000000001e-05 2024-04-26T01:08:19.472 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.472 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.472 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.472 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311836+0000", 2024-04-26T01:08:19.472 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000173349 2024-04-26T01:08:19.472 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.472 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.472 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.473 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311859+0000", 2024-04-26T01:08:19.473 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2504e-05 2024-04-26T01:08:19.473 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.473 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.473 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.473 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322193+0000", 2024-04-26T01:08:19.473 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010334157 2024-04-26T01:08:19.473 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.473 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.473 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.473 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322323+0000", 2024-04-26T01:08:19.473 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013037899999999999 2024-04-26T01:08:19.473 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.473 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.474 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.474 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.474 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.474 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.8s3 32/30 ECSubWrite(tid=1, reqid=client.4521.0:8, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:19.474 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.312999+0000", 2024-04-26T01:08:19.474 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.14478626, 2024-04-26T01:08:19.474 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0094850960000000002, 2024-04-26T01:08:19.474 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.474 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.474 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.474 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.474 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.474 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.475 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312999+0000", 2024-04-26T01:08:19.475 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.475 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.475 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.475 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.475 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312999+0000", 2024-04-26T01:08:19.475 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.475 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.475 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.475 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.475 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313001+0000", 2024-04-26T01:08:19.475 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.21e-06 2024-04-26T01:08:19.475 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.475 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.476 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.476 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313076+0000", 2024-04-26T01:08:19.476 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.5525000000000006e-05 2024-04-26T01:08:19.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.476 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.476 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.476 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313194+0000", 2024-04-26T01:08:19.476 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000118004 2024-04-26T01:08:19.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.476 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.476 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.476 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313238+0000", 2024-04-26T01:08:19.476 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3523999999999999e-05 2024-04-26T01:08:19.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.476 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.477 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.477 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313570+0000", 2024-04-26T01:08:19.477 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033178100000000001 2024-04-26T01:08:19.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.477 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.477 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.477 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313595+0000", 2024-04-26T01:08:19.477 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5181000000000002e-05 2024-04-26T01:08:19.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.477 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.477 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.477 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322175+0000", 2024-04-26T01:08:19.477 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0085803070000000006 2024-04-26T01:08:19.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322484+0000", 2024-04-26T01:08:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000308564 2024-04-26T01:08:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.es2 33/30 ECSubWrite(tid=1, reqid=client.4521.0:23, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.313493+0000", 2024-04-26T01:08:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.144292227000001, 2024-04-26T01:08:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0089452720000000006, 2024-04-26T01:08:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.479 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.479 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.479 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.479 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313493+0000", 2024-04-26T01:08:19.479 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.479 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.479 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313493+0000", 2024-04-26T01:08:19.479 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.479 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.479 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313494+0000", 2024-04-26T01:08:19.480 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5039999999999999e-06 2024-04-26T01:08:19.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.480 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.480 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.480 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313593+0000", 2024-04-26T01:08:19.480 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.9355e-05 2024-04-26T01:08:19.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.480 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.480 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.480 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313756+0000", 2024-04-26T01:08:19.480 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000162762 2024-04-26T01:08:19.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.480 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.480 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313781+0000", 2024-04-26T01:08:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4893e-05 2024-04-26T01:08:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313944+0000", 2024-04-26T01:08:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016297600000000001 2024-04-26T01:08:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313968+0000", 2024-04-26T01:08:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3986999999999999e-05 2024-04-26T01:08:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.482 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322340+0000", 2024-04-26T01:08:19.482 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0083720840000000001 2024-04-26T01:08:19.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.482 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.482 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322438+0000", 2024-04-26T01:08:19.482 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7711e-05 2024-04-26T01:08:19.482 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.482 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.482 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.482 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.2s5 32/30 ECSubWrite(tid=158, reqid=client.4521.0:14, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:19.483 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.317021+0000", 2024-04-26T01:08:19.483 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.140763440000001, 2024-04-26T01:08:19.483 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0053200579999999999, 2024-04-26T01:08:19.483 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.483 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.483 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.483 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.483 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.483 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317021+0000", 2024-04-26T01:08:19.483 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.483 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317021+0000", 2024-04-26T01:08:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317024+0000", 2024-04-26T01:08:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6400000000000001e-06 2024-04-26T01:08:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317075+0000", 2024-04-26T01:08:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0680999999999998e-05 2024-04-26T01:08:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317190+0000", 2024-04-26T01:08:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000114998 2024-04-26T01:08:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317213+0000", 2024-04-26T01:08:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2827e-05 2024-04-26T01:08:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317879+0000", 2024-04-26T01:08:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00066639300000000004 2024-04-26T01:08:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.486 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.486 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.317911+0000", 2024-04-26T01:08:19.486 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.2400000000000001e-05 2024-04-26T01:08:19.486 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.486 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.486 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322261+0000", 2024-04-26T01:08:19.486 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.004349594 2024-04-26T01:08:19.486 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.486 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.486 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322341+0000", 2024-04-26T01:08:19.486 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.0525000000000006e-05 2024-04-26T01:08:19.486 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.486 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.487 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.487 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.487 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.es2 33/30 ECSubWrite(tid=2, reqid=client.4521.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=33'2))", 2024-04-26T01:08:19.487 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.323780+0000", 2024-04-26T01:08:19.487 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.134004587, 2024-04-26T01:08:19.487 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0055927879999999996, 2024-04-26T01:08:19.487 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.487 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.487 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.487 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.487 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.487 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.488 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323780+0000", 2024-04-26T01:08:19.488 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.488 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.488 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.488 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323780+0000", 2024-04-26T01:08:19.488 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.488 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.488 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.488 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323782+0000", 2024-04-26T01:08:19.488 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.717e-06 2024-04-26T01:08:19.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.489 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.489 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.489 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323811+0000", 2024-04-26T01:08:19.489 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8634000000000001e-05 2024-04-26T01:08:19.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.489 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.489 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.489 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323899+0000", 2024-04-26T01:08:19.489 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.8015000000000003e-05 2024-04-26T01:08:19.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.489 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.489 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.489 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324040+0000", 2024-04-26T01:08:19.489 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000141898 2024-04-26T01:08:19.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.490 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.490 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.490 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324324+0000", 2024-04-26T01:08:19.490 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00028301699999999999 2024-04-26T01:08:19.490 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.490 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.490 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.490 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324347+0000", 2024-04-26T01:08:19.490 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3654000000000001e-05 2024-04-26T01:08:19.490 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.490 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.490 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.490 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.329290+0000", 2024-04-26T01:08:19.491 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0049432449999999998 2024-04-26T01:08:19.491 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.491 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.491 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.491 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.329373+0000", 2024-04-26T01:08:19.491 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.2608000000000006e-05 2024-04-26T01:08:19.491 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.491 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.491 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.491 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.491 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.491 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.8s3 33/30 ECSubWrite(tid=2, reqid=client.4521.0:8, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:19.491 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.326417+0000", 2024-04-26T01:08:19.491 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.131367758, 2024-04-26T01:08:19.492 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0057806029999999996, 2024-04-26T01:08:19.492 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.492 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.492 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.492 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.492 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.492 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.492 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326417+0000", 2024-04-26T01:08:19.492 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.492 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.492 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.492 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.492 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326417+0000", 2024-04-26T01:08:19.492 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326419+0000", 2024-04-26T01:08:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.753e-06 2024-04-26T01:08:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326479+0000", 2024-04-26T01:08:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.9800999999999999e-05 2024-04-26T01:08:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326579+0000", 2024-04-26T01:08:19.494 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.9970999999999995e-05 2024-04-26T01:08:19.494 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.494 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.494 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.494 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326602+0000", 2024-04-26T01:08:19.494 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3234000000000002e-05 2024-04-26T01:08:19.494 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.494 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.494 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.494 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327273+0000", 2024-04-26T01:08:19.494 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00067146199999999995 2024-04-26T01:08:19.494 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.494 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.494 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.494 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327298+0000", 2024-04-26T01:08:19.495 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4763999999999999e-05 2024-04-26T01:08:19.495 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.495 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.495 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.495 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.331893+0000", 2024-04-26T01:08:19.495 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0045952379999999998 2024-04-26T01:08:19.495 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.495 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.495 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.495 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.332198+0000", 2024-04-26T01:08:19.495 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00030437999999999998 2024-04-26T01:08:19.495 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.495 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.495 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.496 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.496 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.496 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.6s2 33/30 ECSubWrite(tid=1, reqid=client.4521.0:44, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:19.496 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.349917+0000", 2024-04-26T01:08:19.496 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.107868133, 2024-04-26T01:08:19.496 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0052687769999999997, 2024-04-26T01:08:19.496 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.496 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.496 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.496 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.496 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.496 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.496 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.349917+0000", 2024-04-26T01:08:19.497 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.497 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.497 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.497 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.497 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.349917+0000", 2024-04-26T01:08:19.497 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.497 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.497 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.497 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.497 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.349918+0000", 2024-04-26T01:08:19.497 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.607e-06 2024-04-26T01:08:19.497 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.497 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.497 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.498 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.349961+0000", 2024-04-26T01:08:19.498 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.2870000000000001e-05 2024-04-26T01:08:19.498 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.498 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.498 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.498 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350046+0000", 2024-04-26T01:08:19.498 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.4957999999999995e-05 2024-04-26T01:08:19.498 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.498 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.498 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.498 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350069+0000", 2024-04-26T01:08:19.498 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2574000000000001e-05 2024-04-26T01:08:19.498 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.499 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.499 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.499 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350881+0000", 2024-04-26T01:08:19.499 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00081191100000000001 2024-04-26T01:08:19.499 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.499 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.499 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.499 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350900+0000", 2024-04-26T01:08:19.499 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9559999999999999e-05 2024-04-26T01:08:19.499 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.499 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.499 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.499 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.355103+0000", 2024-04-26T01:08:19.499 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0042028559999999996 2024-04-26T01:08:19.500 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.500 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.500 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.500 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.355185+0000", 2024-04-26T01:08:19.500 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.2441000000000006e-05 2024-04-26T01:08:19.500 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.500 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.500 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.500 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.500 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.500 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/30 ECSubWrite(tid=17, reqid=client.4521.0:224, at_version=36'10, trim_to=0'0, roll_forward_to=33'8))", 2024-04-26T01:08:19.500 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:02.430740+0000", 2024-04-26T01:08:19.500 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 17.027045094999998, 2024-04-26T01:08:19.500 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0054526749999999997, 2024-04-26T01:08:19.501 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.501 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.501 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.501 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.501 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.501 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.501 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.430740+0000", 2024-04-26T01:08:19.501 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.501 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.501 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.501 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.501 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.430740+0000", 2024-04-26T01:08:19.501 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.501 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.502 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.502 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.502 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.430740+0000", 2024-04-26T01:08:19.502 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.4400000000000001e-07 2024-04-26T01:08:19.502 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.502 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.502 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.502 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.430878+0000", 2024-04-26T01:08:19.502 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013799099999999999 2024-04-26T01:08:19.502 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.502 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.502 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.502 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.430978+0000", 2024-04-26T01:08:19.502 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000100188 2024-04-26T01:08:19.503 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.503 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.503 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.503 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.431004+0000", 2024-04-26T01:08:19.503 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5490000000000002e-05 2024-04-26T01:08:19.503 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.503 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.503 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.503 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.431197+0000", 2024-04-26T01:08:19.503 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019329300000000001 2024-04-26T01:08:19.503 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.503 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.503 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.431217+0000", 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.02e-05 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.436087+0000", 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0048693039999999996 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.436192+0000", 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000105765 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.504 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.505 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.505 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.2s5 36/30 ECSubWrite(tid=184, reqid=client.4521.0:225, at_version=36'13, trim_to=0'0, roll_forward_to=33'10))", 2024-04-26T01:08:19.505 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:02.437781+0000", 2024-04-26T01:08:19.505 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 17.020004309000001, 2024-04-26T01:08:19.505 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0053170939999999996, 2024-04-26T01:08:19.505 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.505 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.505 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.505 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.505 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.505 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.505 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.437781+0000", 2024-04-26T01:08:19.505 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.505 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.506 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.506 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.506 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.437781+0000", 2024-04-26T01:08:19.506 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.506 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.506 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.506 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.506 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.437781+0000", 2024-04-26T01:08:19.506 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2299999999999999e-07 2024-04-26T01:08:19.506 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.506 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.506 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.506 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.437843+0000", 2024-04-26T01:08:19.507 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.1851000000000001e-05 2024-04-26T01:08:19.507 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.507 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.507 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.507 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.437990+0000", 2024-04-26T01:08:19.507 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014688799999999999 2024-04-26T01:08:19.507 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.507 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.507 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.507 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.438017+0000", 2024-04-26T01:08:19.507 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7146999999999999e-05 2024-04-26T01:08:19.507 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.507 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.507 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.507 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.438295+0000", 2024-04-26T01:08:19.508 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00027786999999999999 2024-04-26T01:08:19.508 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.508 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.508 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.508 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.438321+0000", 2024-04-26T01:08:19.508 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5431000000000001e-05 2024-04-26T01:08:19.508 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.508 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.508 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.508 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.443000+0000", 2024-04-26T01:08:19.508 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0046792759999999996 2024-04-26T01:08:19.508 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.508 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.508 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.509 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.443098+0000", 2024-04-26T01:08:19.509 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7707999999999994e-05 2024-04-26T01:08:19.509 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.509 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.509 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.509 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.509 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.509 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.2s5 36/30 ECSubWrite(tid=185, reqid=client.4521.0:225, at_version=0'0, trim_to=0'0, roll_forward_to=36'13))", 2024-04-26T01:08:19.509 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:02.445957+0000", 2024-04-26T01:08:19.509 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 17.011828222999998, 2024-04-26T01:08:19.509 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0052070720000000001, 2024-04-26T01:08:19.509 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.509 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.509 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.509 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.510 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.510 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.510 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.445957+0000", 2024-04-26T01:08:19.510 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.510 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.510 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.510 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.510 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.445957+0000", 2024-04-26T01:08:19.510 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.510 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.510 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.510 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.510 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.445958+0000", 2024-04-26T01:08:19.510 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6130000000000001e-06 2024-04-26T01:08:19.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.511 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.511 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.511 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.445979+0000", 2024-04-26T01:08:19.511 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0893000000000001e-05 2024-04-26T01:08:19.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.511 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.511 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.511 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.446058+0000", 2024-04-26T01:08:19.511 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8450999999999998e-05 2024-04-26T01:08:19.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.511 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.511 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.511 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.446073+0000", 2024-04-26T01:08:19.512 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5167000000000001e-05 2024-04-26T01:08:19.512 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.512 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.512 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.512 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.446249+0000", 2024-04-26T01:08:19.512 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017641900000000001 2024-04-26T01:08:19.512 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.512 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.512 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.512 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.446281+0000", 2024-04-26T01:08:19.512 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.2123999999999999e-05 2024-04-26T01:08:19.512 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.512 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.512 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.512 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.451082+0000", 2024-04-26T01:08:19.513 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0048012009999999997 2024-04-26T01:08:19.513 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.513 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.513 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.513 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:02.451164+0000", 2024-04-26T01:08:19.513 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1204000000000004e-05 2024-04-26T01:08:19.513 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.513 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.513 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.513 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.513 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.513 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.6s2 43/30 ECSubWrite(tid=32, reqid=client.4521.0:391, at_version=43'17, trim_to=0'0, roll_forward_to=43'14))", 2024-04-26T01:08:19.513 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:09.466971+0000", 2024-04-26T01:08:19.513 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.9908140430000003, 2024-04-26T01:08:19.514 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0053763439999999999, 2024-04-26T01:08:19.514 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.514 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.514 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.514 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.514 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.514 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.514 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:09.466971+0000", 2024-04-26T01:08:19.514 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.514 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.514 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.514 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.514 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:09.466971+0000", 2024-04-26T01:08:19.514 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:09.466972+0000", 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5e-06 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:09.467014+0000", 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.1507000000000001e-05 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:09.467127+0000", 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000113591 2024-04-26T01:08:19.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.516 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.516 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.516 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:09.467151+0000", 2024-04-26T01:08:19.516 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3567e-05 2024-04-26T01:08:19.516 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.516 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.516 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.516 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:09.468102+0000", 2024-04-26T01:08:19.516 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00095093700000000001 2024-04-26T01:08:19.516 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.516 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.516 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.516 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:09.468127+0000", 2024-04-26T01:08:19.516 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5293999999999999e-05 2024-04-26T01:08:19.517 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.517 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.517 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.517 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:09.472268+0000", 2024-04-26T01:08:19.517 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0041405849999999996 2024-04-26T01:08:19.517 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.517 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.517 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.517 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:09.472347+0000", 2024-04-26T01:08:19.517 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.9363000000000006e-05 2024-04-26T01:08:19.517 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.517 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.517 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.518 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.518 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.518 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 50/30 ECSubWrite(tid=29, reqid=osd.8.0:29, at_version=50'18, trim_to=0'0, roll_forward_to=47'16))", 2024-04-26T01:08:19.518 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.040723+0000", 2024-04-26T01:08:19.518 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 3.4170619869999999, 2024-04-26T01:08:19.518 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012574701000000001, 2024-04-26T01:08:19.518 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.518 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.518 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.518 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.518 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.518 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.518 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040723+0000", 2024-04-26T01:08:19.518 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040723+0000", 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040724+0000", 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.37e-07 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040778+0000", 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.4453000000000002e-05 2024-04-26T01:08:19.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.520 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.520 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.520 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040860+0000", 2024-04-26T01:08:19.520 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.2060999999999999e-05 2024-04-26T01:08:19.520 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.520 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.520 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.520 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040883+0000", 2024-04-26T01:08:19.520 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2994e-05 2024-04-26T01:08:19.520 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.520 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.520 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.520 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.041058+0000", 2024-04-26T01:08:19.520 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017496999999999999 2024-04-26T01:08:19.520 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.521 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.521 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.521 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.041082+0000", 2024-04-26T01:08:19.521 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.393e-05 2024-04-26T01:08:19.521 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.521 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.521 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.521 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053181+0000", 2024-04-26T01:08:19.521 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012099085000000001 2024-04-26T01:08:19.521 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.521 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.521 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.521 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053298+0000", 2024-04-26T01:08:19.521 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000116471 2024-04-26T01:08:19.522 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.522 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.522 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.522 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.522 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.522 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 50/30 ECSubWrite(tid=214, reqid=osd.4.0:214, at_version=50'22, trim_to=0'0, roll_forward_to=45'20))", 2024-04-26T01:08:19.522 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.047877+0000", 2024-04-26T01:08:19.522 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 3.4099073230000001, 2024-04-26T01:08:19.522 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010720746, 2024-04-26T01:08:19.522 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.522 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.522 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.522 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.523 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.523 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.523 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047877+0000", 2024-04-26T01:08:19.523 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.523 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.523 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.523 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.523 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047877+0000", 2024-04-26T01:08:19.523 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.523 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.523 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.523 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.523 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047878+0000", 2024-04-26T01:08:19.523 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.6699999999999999e-07 2024-04-26T01:08:19.523 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047930+0000", 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.1669999999999998e-05 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048154+0000", 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022397399999999999 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048176+0000", 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2734e-05 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.524 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.525 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.525 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048452+0000", 2024-04-26T01:08:19.525 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00027581799999999999 2024-04-26T01:08:19.525 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.525 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.525 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.525 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048478+0000", 2024-04-26T01:08:19.525 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5723000000000001e-05 2024-04-26T01:08:19.525 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.525 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.525 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.525 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058506+0000", 2024-04-26T01:08:19.525 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010027635999999999 2024-04-26T01:08:19.525 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.526 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.526 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.526 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058598+0000", 2024-04-26T01:08:19.526 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2724000000000002e-05 2024-04-26T01:08:19.526 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.526 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.526 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.526 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.526 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.526 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.6s2 50/30 ECSubWrite(tid=41, reqid=osd.0.0:41, at_version=50'24, trim_to=0'0, roll_forward_to=45'22))", 2024-04-26T01:08:19.526 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.052765+0000", 2024-04-26T01:08:19.526 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 3.4050195790000002, 2024-04-26T01:08:19.527 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0080835920000000006, 2024-04-26T01:08:19.527 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.527 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.527 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.527 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.527 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.527 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.527 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052765+0000", 2024-04-26T01:08:19.527 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.527 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.527 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.527 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.527 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052765+0000", 2024-04-26T01:08:19.527 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.527 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.528 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.528 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.528 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052766+0000", 2024-04-26T01:08:19.528 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1999999999999999e-07 2024-04-26T01:08:19.528 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.528 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.528 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.528 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052820+0000", 2024-04-26T01:08:19.528 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.4441000000000003e-05 2024-04-26T01:08:19.528 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.528 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.528 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.528 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052916+0000", 2024-04-26T01:08:19.528 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5597e-05 2024-04-26T01:08:19.529 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.529 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.529 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.529 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052940+0000", 2024-04-26T01:08:19.529 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3626e-05 2024-04-26T01:08:19.529 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.529 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.529 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.529 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053100+0000", 2024-04-26T01:08:19.529 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015994800000000001 2024-04-26T01:08:19.529 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.529 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.529 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.530 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053126+0000", 2024-04-26T01:08:19.530 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6393e-05 2024-04-26T01:08:19.530 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.530 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.530 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.530 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060748+0000", 2024-04-26T01:08:19.530 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.007622523 2024-04-26T01:08:19.530 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.530 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.530 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.530 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060849+0000", 2024-04-26T01:08:19.530 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000100344 2024-04-26T01:08:19.531 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.531 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.531 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.531 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.531 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.531 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 50/30 ECSubWrite(tid=30, reqid=osd.8.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=50'18))", 2024-04-26T01:08:19.531 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.054231+0000", 2024-04-26T01:08:19.531 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 3.4035538760000001, 2024-04-26T01:08:19.531 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0065502090000000004, 2024-04-26T01:08:19.531 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.531 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.531 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.531 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.531 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.532 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.532 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.054231+0000", 2024-04-26T01:08:19.532 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.532 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.532 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.532 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.532 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.054231+0000", 2024-04-26T01:08:19.532 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.532 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.532 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.532 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.532 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.054233+0000", 2024-04-26T01:08:19.532 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.734e-06 2024-04-26T01:08:19.532 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.533 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.533 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.533 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.054918+0000", 2024-04-26T01:08:19.533 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00068537299999999997 2024-04-26T01:08:19.533 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.533 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.533 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.533 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055321+0000", 2024-04-26T01:08:19.533 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00040306400000000002 2024-04-26T01:08:19.533 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.533 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.533 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.533 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055368+0000", 2024-04-26T01:08:19.534 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.7277999999999997e-05 2024-04-26T01:08:19.534 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.534 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.534 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.534 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.056224+0000", 2024-04-26T01:08:19.534 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00085513600000000005 2024-04-26T01:08:19.534 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.534 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.534 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.534 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.056250+0000", 2024-04-26T01:08:19.534 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6554e-05 2024-04-26T01:08:19.534 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.534 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.535 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.535 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060684+0000", 2024-04-26T01:08:19.535 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0044339560000000002 2024-04-26T01:08:19.535 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.535 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.535 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.535 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060781+0000", 2024-04-26T01:08:19.535 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7114000000000006e-05 2024-04-26T01:08:19.535 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.535 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.535 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.535 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.535 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.535 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.es2 50/44 ECSubWrite(tid=51, reqid=osd.2.0:51, at_version=50'40, trim_to=0'0, roll_forward_to=45'38))", 2024-04-26T01:08:19.536 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.058416+0000", 2024-04-26T01:08:19.536 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 3.3993688209999999, 2024-04-26T01:08:19.536 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0062012100000000004, 2024-04-26T01:08:19.536 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.536 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.536 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.536 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.536 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.536 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.536 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058416+0000", 2024-04-26T01:08:19.536 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.536 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.536 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.536 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058416+0000", 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058417+0000", 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.44e-07 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058472+0000", 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.5179999999999997e-05 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.537 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058579+0000", 2024-04-26T01:08:19.538 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000107113 2024-04-26T01:08:19.538 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.538 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.538 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.538 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058602+0000", 2024-04-26T01:08:19.538 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3294000000000001e-05 2024-04-26T01:08:19.538 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.538 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.538 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.538 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.059444+0000", 2024-04-26T01:08:19.538 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00084185300000000005 2024-04-26T01:08:19.538 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.538 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.539 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.539 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.059470+0000", 2024-04-26T01:08:19.539 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6444e-05 2024-04-26T01:08:19.539 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.539 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.539 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.539 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.064521+0000", 2024-04-26T01:08:19.539 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.005050868 2024-04-26T01:08:19.539 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.539 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.539 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.539 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.064617+0000", 2024-04-26T01:08:19.539 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5914000000000004e-05 2024-04-26T01:08:19.539 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.540 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.540 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.540 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.540 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.540 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s3 50/30 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=50'32, trim_to=0'0, roll_forward_to=45'30))", 2024-04-26T01:08:19.540 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.061492+0000", 2024-04-26T01:08:19.540 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 3.3962926879999999, 2024-04-26T01:08:19.540 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0050850160000000004, 2024-04-26T01:08:19.540 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.540 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.540 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.540 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.540 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.540 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061492+0000", 2024-04-26T01:08:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061492+0000", 2024-04-26T01:08:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061493+0000", 2024-04-26T01:08:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.2399999999999998e-07 2024-04-26T01:08:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061547+0000", 2024-04-26T01:08:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.401e-05 2024-04-26T01:08:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061670+0000", 2024-04-26T01:08:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012341699999999999 2024-04-26T01:08:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061692+0000", 2024-04-26T01:08:19.543 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1500000000000001e-05 2024-04-26T01:08:19.543 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.543 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.543 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.543 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061914+0000", 2024-04-26T01:08:19.543 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022260100000000001 2024-04-26T01:08:19.543 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.543 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.543 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.543 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061941+0000", 2024-04-26T01:08:19.543 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7169999999999999e-05 2024-04-26T01:08:19.543 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.543 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.543 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.543 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066473+0000", 2024-04-26T01:08:19.544 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0045316760000000001 2024-04-26T01:08:19.544 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.544 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.544 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.544 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066577+0000", 2024-04-26T01:08:19.544 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00010401800000000001 2024-04-26T01:08:19.544 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.544 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.544 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.544 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.544 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.544 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.2s5 50/44 ECSubWrite(tid=212, reqid=osd.13.0:212, at_version=50'24, trim_to=0'0, roll_forward_to=45'22))", 2024-04-26T01:08:19.544 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.062403+0000", 2024-04-26T01:08:19.544 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 3.3953817900000001, 2024-04-26T01:08:19.544 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0059074710000000001, 2024-04-26T01:08:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.062403+0000", 2024-04-26T01:08:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.062403+0000", 2024-04-26T01:08:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.546 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:19.546 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.062404+0000", 2024-04-26T01:08:19.546 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.06e-07 2024-04-26T01:08:19.546 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.546 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.546 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:19.546 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.062453+0000", 2024-04-26T01:08:19.546 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.9357000000000002e-05 2024-04-26T01:08:19.546 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.546 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.546 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:19.546 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.062738+0000", 2024-04-26T01:08:19.546 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00028555099999999999 2024-04-26T01:08:19.547 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.547 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.547 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:19.547 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.062760+0000", 2024-04-26T01:08:19.547 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2050000000000001e-05 2024-04-26T01:08:19.547 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.547 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.547 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:19.547 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063197+0000", 2024-04-26T01:08:19.547 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000436205 2024-04-26T01:08:19.547 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.547 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.547 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:19.547 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063235+0000", 2024-04-26T01:08:19.548 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.8383999999999997e-05 2024-04-26T01:08:19.548 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.548 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.548 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:19.548 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.068214+0000", 2024-04-26T01:08:19.548 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0049789040000000001 2024-04-26T01:08:19.548 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:19.548 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:19.548 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:19.548 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.068310+0000", 2024-04-26T01:08:19.548 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6514000000000005e-05 2024-04-26T01:08:19.548 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.548 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.548 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.548 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:19.549 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:19.549 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:19.550 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:08:19.551 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:19.548+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:19.651 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:19.645+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:19.730 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:19.730 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:19.730 INFO:teuthology.orchestra.run.smithi012.stdout: "num_ops": 0 2024-04-26T01:08:19.730 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:19.745 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:08:19.751 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:19.752+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:19.853 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:19.852+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:19.913 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:19.913 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:19.913 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:08:19.914 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:08:19.914 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:19.927 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:08:19.953 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:19.952+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:20.052 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:08:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:08:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:08:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:7 3.8s0 3:1b6b0095:::smithi16428812-3:head [append 0~49152,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.302696+0000", 2024-04-26T01:08:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.747581498999999, 2024-04-26T01:08:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.016152680999999999, 2024-04-26T01:08:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 7 2024-04-26T01:08:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.302696+0000", 2024-04-26T01:08:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.302696+0000", 2024-04-26T01:08:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.302704+0000", 2024-04-26T01:08:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1130000000000001e-06 2024-04-26T01:08:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303173+0000", 2024-04-26T01:08:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000469109 2024-04-26T01:08:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303207+0000", 2024-04-26T01:08:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.3937999999999999e-05 2024-04-26T01:08:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303229+0000", 2024-04-26T01:08:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1573000000000002e-05 2024-04-26T01:08:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304061+0000", 2024-04-26T01:08:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00083202500000000002 2024-04-26T01:08:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304318+0000", 2024-04-26T01:08:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025779299999999998 2024-04-26T01:08:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.306191+0000", 2024-04-26T01:08:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0018724010000000001 2024-04-26T01:08:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318520+0000", 2024-04-26T01:08:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012329028000000001 2024-04-26T01:08:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318628+0000", 2024-04-26T01:08:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00010778800000000001 2024-04-26T01:08:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318849+0000", 2024-04-26T01:08:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022091299999999999 2024-04-26T01:08:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:8 3.8s0 3:1b6b0095:::smithi16428812-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.303305+0000", 2024-04-26T01:08:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.746972207999999, 2024-04-26T01:08:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.020292667, 2024-04-26T01:08:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 8 2024-04-26T01:08:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303305+0000", 2024-04-26T01:08:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303305+0000", 2024-04-26T01:08:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303307+0000", 2024-04-26T01:08:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7770000000000001e-06 2024-04-26T01:08:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303320+0000", 2024-04-26T01:08:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3086e-05 2024-04-26T01:08:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303343+0000", 2024-04-26T01:08:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3323999999999999e-05 2024-04-26T01:08:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303387+0000", 2024-04-26T01:08:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3741e-05 2024-04-26T01:08:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.307388+0000", 2024-04-26T01:08:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0040010430000000001 2024-04-26T01:08:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.307645+0000", 2024-04-26T01:08:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025681400000000001 2024-04-26T01:08:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311893+0000", 2024-04-26T01:08:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.004247626 2024-04-26T01:08:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318857+0000", 2024-04-26T01:08:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0069646760000000004 2024-04-26T01:08:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323372+0000", 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0045143329999999997 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323598+0000", 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000226247 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:9 3.8s0 3:1b6b0095:::smithi16428812-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.303410+0000", 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.746867127000002, 2024-04-26T01:08:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.028697769000000001, 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 9 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303410+0000", 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303410+0000", 2024-04-26T01:08:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303412+0000", 2024-04-26T01:08:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.137e-06 2024-04-26T01:08:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303420+0000", 2024-04-26T01:08:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.3540000000000002e-06 2024-04-26T01:08:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303439+0000", 2024-04-26T01:08:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9227000000000001e-05 2024-04-26T01:08:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303455+0000", 2024-04-26T01:08:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5852999999999999e-05 2024-04-26T01:08:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312566+0000", 2024-04-26T01:08:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0091115209999999992 2024-04-26T01:08:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312793+0000", 2024-04-26T01:08:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022636999999999999 2024-04-26T01:08:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325814+0000", 2024-04-26T01:08:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013020893 2024-04-26T01:08:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326042+0000", 2024-04-26T01:08:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022811899999999999 2024-04-26T01:08:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.332108+0000", 2024-04-26T01:08:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0060662950000000002 2024-04-26T01:08:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 33/30 ECSubWrite(tid=4, reqid=client.4521.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.323649+0000", 2024-04-26T01:08:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.726628284, 2024-04-26T01:08:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0067605859999999999, 2024-04-26T01:08:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323649+0000", 2024-04-26T01:08:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323649+0000", 2024-04-26T01:08:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323651+0000", 2024-04-26T01:08:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7430000000000001e-06 2024-04-26T01:08:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323677+0000", 2024-04-26T01:08:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5711000000000002e-05 2024-04-26T01:08:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323735+0000", 2024-04-26T01:08:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.8619999999999998e-05 2024-04-26T01:08:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323758+0000", 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2351e-05 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323954+0000", 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019601199999999999 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323978+0000", 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4284e-05 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.330333+0000", 2024-04-26T01:08:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0063547940000000004 2024-04-26T01:08:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.330410+0000", 2024-04-26T01:08:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.7070999999999994e-05 2024-04-26T01:08:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s5 33/30 ECSubWrite(tid=8, reqid=client.4521.0:37, at_version=33'3, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.324674+0000", 2024-04-26T01:08:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.725603841000002, 2024-04-26T01:08:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0070497340000000002, 2024-04-26T01:08:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324674+0000", 2024-04-26T01:08:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324674+0000", 2024-04-26T01:08:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324675+0000", 2024-04-26T01:08:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5370000000000001e-06 2024-04-26T01:08:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.324970+0000", 2024-04-26T01:08:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00029448700000000001 2024-04-26T01:08:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325122+0000", 2024-04-26T01:08:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015275600000000001 2024-04-26T01:08:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325143+0000", 2024-04-26T01:08:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0985999999999999e-05 2024-04-26T01:08:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325543+0000", 2024-04-26T01:08:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000399486 2024-04-26T01:08:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325574+0000", 2024-04-26T01:08:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.1186999999999998e-05 2024-04-26T01:08:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.331600+0000", 2024-04-26T01:08:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0060261999999999998 2024-04-26T01:08:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.331723+0000", 2024-04-26T01:08:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012309500000000001 2024-04-26T01:08:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:49 3.8s0 3:11b43273:::smithi16428812-17:head [append 0~278528,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.342928+0000", 2024-04-26T01:08:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.707349202, 2024-04-26T01:08:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0099803930000000006, 2024-04-26T01:08:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 49 2024-04-26T01:08:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.342928+0000", 2024-04-26T01:08:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.342928+0000", 2024-04-26T01:08:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.342931+0000", 2024-04-26T01:08:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.0000000000000001e-06 2024-04-26T01:08:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.343938+0000", 2024-04-26T01:08:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0010073129999999999 2024-04-26T01:08:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.343964+0000", 2024-04-26T01:08:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5936999999999999e-05 2024-04-26T01:08:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.343978+0000", 2024-04-26T01:08:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3481e-05 2024-04-26T01:08:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344123+0000", 2024-04-26T01:08:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014518199999999999 2024-04-26T01:08:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344393+0000", 2024-04-26T01:08:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00026982599999999997 2024-04-26T01:08:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.345686+0000", 2024-04-26T01:08:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001292774 2024-04-26T01:08:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.351912+0000", 2024-04-26T01:08:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.006225904 2024-04-26T01:08:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.352571+0000", 2024-04-26T01:08:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00065908000000000004 2024-04-26T01:08:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.352909+0000", 2024-04-26T01:08:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033789600000000002 2024-04-26T01:08:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:50 3.8s0 3:11b43273:::smithi16428812-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.344014+0000", 2024-04-26T01:08:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.706263428, 2024-04-26T01:08:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013017295999999999, 2024-04-26T01:08:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 50 2024-04-26T01:08:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344014+0000", 2024-04-26T01:08:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344014+0000", 2024-04-26T01:08:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344015+0000", 2024-04-26T01:08:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.437e-06 2024-04-26T01:08:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344022+0000", 2024-04-26T01:08:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.9970000000000001e-06 2024-04-26T01:08:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344034+0000", 2024-04-26T01:08:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1817000000000001e-05 2024-04-26T01:08:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344043+0000", 2024-04-26T01:08:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.8060000000000001e-06 2024-04-26T01:08:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.347851+0000", 2024-04-26T01:08:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0038083570000000001 2024-04-26T01:08:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.348142+0000", 2024-04-26T01:08:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00029042400000000001 2024-04-26T01:08:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350786+0000", 2024-04-26T01:08:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002644202 2024-04-26T01:08:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.352937+0000", 2024-04-26T01:08:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0021514910000000002 2024-04-26T01:08:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.356940+0000", 2024-04-26T01:08:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0040022740000000001 2024-04-26T01:08:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357031+0000", 2024-04-26T01:08:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.1490999999999995e-05 2024-04-26T01:08:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:51 3.8s0 3:11b43273:::smithi16428812-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.344056+0000", 2024-04-26T01:08:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.706221416999998, 2024-04-26T01:08:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.01557094, 2024-04-26T01:08:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 51 2024-04-26T01:08:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344056+0000", 2024-04-26T01:08:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344056+0000", 2024-04-26T01:08:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344057+0000", 2024-04-26T01:08:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1699999999999997e-07 2024-04-26T01:08:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344060+0000", 2024-04-26T01:08:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.703e-06 2024-04-26T01:08:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344071+0000", 2024-04-26T01:08:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.057e-05 2024-04-26T01:08:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344085+0000", 2024-04-26T01:08:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3997e-05 2024-04-26T01:08:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.351519+0000", 2024-04-26T01:08:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0074343079999999997 2024-04-26T01:08:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.351770+0000", 2024-04-26T01:08:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025095500000000003 2024-04-26T01:08:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357516+0000", 2024-04-26T01:08:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0057452919999999999 2024-04-26T01:08:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357612+0000", 2024-04-26T01:08:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6444e-05 2024-04-26T01:08:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.359627+0000", 2024-04-26T01:08:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002014854 2024-04-26T01:08:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 38/30 ECSubWrite(tid=85, reqid=osd.4.0:85, at_version=38'19, trim_to=0'0, roll_forward_to=38'15))", 2024-04-26T01:08:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.432265+0000", 2024-04-26T01:08:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 15.618012447, 2024-04-26T01:08:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011946251999999999, 2024-04-26T01:08:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432265+0000", 2024-04-26T01:08:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432265+0000", 2024-04-26T01:08:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432266+0000", 2024-04-26T01:08:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5900000000000005e-07 2024-04-26T01:08:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432295+0000", 2024-04-26T01:08:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9374999999999999e-05 2024-04-26T01:08:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441494+0000", 2024-04-26T01:08:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0091991369999999996 2024-04-26T01:08:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441529+0000", 2024-04-26T01:08:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.4677e-05 2024-04-26T01:08:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441702+0000", 2024-04-26T01:08:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017256500000000001 2024-04-26T01:08:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441731+0000", 2024-04-26T01:08:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8897000000000001e-05 2024-04-26T01:08:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.444168+0000", 2024-04-26T01:08:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0024374930000000002 2024-04-26T01:08:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.444211+0000", 2024-04-26T01:08:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3149000000000003e-05 2024-04-26T01:08:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 38/30 ECSubWrite(tid=87, reqid=client.4521.0:267, at_version=0'0, trim_to=0'0, roll_forward_to=38'21))", 2024-04-26T01:08:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.447653+0000", 2024-04-26T01:08:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 15.602624047000001, 2024-04-26T01:08:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.55546236199999999, 2024-04-26T01:08:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.447653+0000", 2024-04-26T01:08:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.447653+0000", 2024-04-26T01:08:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.447654+0000", 2024-04-26T01:08:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.8299999999999996e-07 2024-04-26T01:08:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.447665+0000", 2024-04-26T01:08:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0539e-05 2024-04-26T01:08:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.001792+0000", 2024-04-26T01:08:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.55412781 2024-04-26T01:08:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.001811+0000", 2024-04-26T01:08:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8236999999999998e-05 2024-04-26T01:08:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.001927+0000", 2024-04-26T01:08:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000116582 2024-04-26T01:08:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.001939+0000", 2024-04-26T01:08:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2312e-05 2024-04-26T01:08:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.003044+0000", 2024-04-26T01:08:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0011047819999999999 2024-04-26T01:08:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.003116+0000", 2024-04-26T01:08:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1416999999999997e-05 2024-04-26T01:08:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:288 3.8s0 3:1b6b0095:::smithi16428812-3:head [delete] snapc 4=[2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.532919+0000", 2024-04-26T01:08:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 15.517357863000001, 2024-04-26T01:08:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0091303889999999992, 2024-04-26T01:08:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 288 2024-04-26T01:08:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532919+0000", 2024-04-26T01:08:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532919+0000", 2024-04-26T01:08:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532921+0000", 2024-04-26T01:08:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8810000000000001e-06 2024-04-26T01:08:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532929+0000", 2024-04-26T01:08:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8599999999999993e-06 2024-04-26T01:08:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532950+0000", 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0738999999999999e-05 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532960+0000", 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5990000000000008e-06 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.533062+0000", 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000102402 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.533241+0000", 2024-04-26T01:08:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017890899999999999 2024-04-26T01:08:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.534156+0000", 2024-04-26T01:08:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00091496199999999996 2024-04-26T01:08:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.535119+0000", 2024-04-26T01:08:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00096345000000000003 2024-04-26T01:08:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.541980+0000", 2024-04-26T01:08:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0068602719999999997 2024-04-26T01:08:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.542050+0000", 2024-04-26T01:08:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.0314999999999993e-05 2024-04-26T01:08:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:290 3.8s0 3:11b43273:::smithi16428812-17:head [read 0~1294336,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.545945+0000", 2024-04-26T01:08:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 15.504332454, 2024-04-26T01:08:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0074298849999999998, 2024-04-26T01:08:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 290 2024-04-26T01:08:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545945+0000", 2024-04-26T01:08:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545945+0000", 2024-04-26T01:08:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545946+0000", 2024-04-26T01:08:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.0500000000000003e-07 2024-04-26T01:08:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545951+0000", 2024-04-26T01:08:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0429999999999998e-06 2024-04-26T01:08:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545960+0000", 2024-04-26T01:08:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.3859999999999996e-06 2024-04-26T01:08:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545972+0000", 2024-04-26T01:08:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2234e-05 2024-04-26T01:08:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.547938+0000", 2024-04-26T01:08:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001965989 2024-04-26T01:08:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.548007+0000", 2024-04-26T01:08:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.9151000000000003e-05 2024-04-26T01:08:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.553375+0000", 2024-04-26T01:08:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0053673770000000004 2024-04-26T01:08:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.cs6 50/44 ECSubWrite(tid=22, reqid=osd.9.0:22, at_version=50'37, trim_to=0'0, roll_forward_to=47'35))", 2024-04-26T01:08:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.057328+0000", 2024-04-26T01:08:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 3.9929496950000001, 2024-04-26T01:08:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010531041999999999, 2024-04-26T01:08:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057328+0000", 2024-04-26T01:08:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057328+0000", 2024-04-26T01:08:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057328+0000", 2024-04-26T01:08:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.4700000000000001e-07 2024-04-26T01:08:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057387+0000", 2024-04-26T01:08:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.8983000000000001e-05 2024-04-26T01:08:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057937+0000", 2024-04-26T01:08:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00054932600000000005 2024-04-26T01:08:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057963+0000", 2024-04-26T01:08:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6794000000000001e-05 2024-04-26T01:08:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058163+0000", 2024-04-26T01:08:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000199834 2024-04-26T01:08:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058192+0000", 2024-04-26T01:08:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.921e-05 2024-04-26T01:08:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.067751+0000", 2024-04-26T01:08:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0095586939999999995 2024-04-26T01:08:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.067859+0000", 2024-04-26T01:08:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00010755399999999999 2024-04-26T01:08:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/44 ECSubWrite(tid=215, reqid=client.4521.0:490, at_version=50'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-26T01:08:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.059123+0000", 2024-04-26T01:08:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 3.9911541339999999, 2024-04-26T01:08:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.014980383999999999, 2024-04-26T01:08:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.059123+0000", 2024-04-26T01:08:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.059123+0000", 2024-04-26T01:08:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.059124+0000", 2024-04-26T01:08:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0839999999999999e-06 2024-04-26T01:08:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.059964+0000", 2024-04-26T01:08:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00083960600000000001 2024-04-26T01:08:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066110+0000", 2024-04-26T01:08:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0061464900000000001 2024-04-26T01:08:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066137+0000", 2024-04-26T01:08:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6307000000000001e-05 2024-04-26T01:08:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066336+0000", 2024-04-26T01:08:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019909100000000001 2024-04-26T01:08:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066365+0000", 2024-04-26T01:08:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9490000000000001e-05 2024-04-26T01:08:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.073999+0000", 2024-04-26T01:08:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0076337089999999998 2024-04-26T01:08:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.074104+0000", 2024-04-26T01:08:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000104607 2024-04-26T01:08:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 50/30 ECSubWriteReply(tid=41, last_complete=50'26, committed=1, applied=1))", 2024-04-26T01:08:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.060433+0000", 2024-04-26T01:08:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 3.9898446650000001, 2024-04-26T01:08:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0074890549999999997, 2024-04-26T01:08:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060433+0000", 2024-04-26T01:08:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060433+0000", 2024-04-26T01:08:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060433+0000", 2024-04-26T01:08:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.5000000000000002e-07 2024-04-26T01:08:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060441+0000", 2024-04-26T01:08:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.233e-06 2024-04-26T01:08:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.062232+0000", 2024-04-26T01:08:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001791441 2024-04-26T01:08:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.062254+0000", 2024-04-26T01:08:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1846999999999999e-05 2024-04-26T01:08:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066505+0000", 2024-04-26T01:08:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0042505879999999996 2024-04-26T01:08:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.067922+0000", 2024-04-26T01:08:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0014171959999999999 2024-04-26T01:08:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/44 ECSubWrite(tid=218, reqid=osd.4.0:218, at_version=50'34, trim_to=0'0, roll_forward_to=47'30))", 2024-04-26T01:08:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.065908+0000", 2024-04-26T01:08:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 3.9843689380000002, 2024-04-26T01:08:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010766286, 2024-04-26T01:08:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065908+0000", 2024-04-26T01:08:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065908+0000", 2024-04-26T01:08:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065911+0000", 2024-04-26T01:08:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5900000000000002e-06 2024-04-26T01:08:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065971+0000", 2024-04-26T01:08:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.0152999999999998e-05 2024-04-26T01:08:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066224+0000", 2024-04-26T01:08:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025274800000000001 2024-04-26T01:08:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066242+0000", 2024-04-26T01:08:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8499999999999999e-05 2024-04-26T01:08:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.068803+0000", 2024-04-26T01:08:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002560414 2024-04-26T01:08:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.068831+0000", 2024-04-26T01:08:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8586999999999999e-05 2024-04-26T01:08:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076582+0000", 2024-04-26T01:08:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0077506160000000001 2024-04-26T01:08:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076675+0000", 2024-04-26T01:08:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2677999999999996e-05 2024-04-26T01:08:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 50/44 ECSubWrite(tid=219, reqid=osd.4.0:219, at_version=50'58, trim_to=0'0, roll_forward_to=47'56))", 2024-04-26T01:08:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.065984+0000", 2024-04-26T01:08:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 3.9842937149999997, 2024-04-26T01:08:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.018105768000000001, 2024-04-26T01:08:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065984+0000", 2024-04-26T01:08:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065984+0000", 2024-04-26T01:08:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065984+0000", 2024-04-26T01:08:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.7700000000000004e-07 2024-04-26T01:08:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066035+0000", 2024-04-26T01:08:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0470000000000003e-05 2024-04-26T01:08:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077389+0000", 2024-04-26T01:08:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011354326 2024-04-26T01:08:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077419+0000", 2024-04-26T01:08:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.0006e-05 2024-04-26T01:08:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077629+0000", 2024-04-26T01:08:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000209471 2024-04-26T01:08:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077685+0000", 2024-04-26T01:08:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.6674e-05 2024-04-26T01:08:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.083973+0000", 2024-04-26T01:08:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0062876870000000001 2024-04-26T01:08:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.084089+0000", 2024-04-26T01:08:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000116357 2024-04-26T01:08:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/30 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=50'32, trim_to=0'0, roll_forward_to=45'30))", 2024-04-26T01:08:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.066181+0000", 2024-04-26T01:08:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 3.984096294, 2024-04-26T01:08:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011395499, 2024-04-26T01:08:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066181+0000", 2024-04-26T01:08:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066181+0000", 2024-04-26T01:08:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066183+0000", 2024-04-26T01:08:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.767e-06 2024-04-26T01:08:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066240+0000", 2024-04-26T01:08:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.7099999999999999e-05 2024-04-26T01:08:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066338+0000", 2024-04-26T01:08:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7738000000000005e-05 2024-04-26T01:08:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066361+0000", 2024-04-26T01:08:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3615999999999999e-05 2024-04-26T01:08:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.070266+0000", 2024-04-26T01:08:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0039045659999999999 2024-04-26T01:08:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.070819+0000", 2024-04-26T01:08:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00055286599999999995 2024-04-26T01:08:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077394+0000", 2024-04-26T01:08:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0065747990000000001 2024-04-26T01:08:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077577+0000", 2024-04-26T01:08:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00018304700000000001 2024-04-26T01:08:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/44 ECSubWrite(tid=221, reqid=osd.4.0:218, at_version=0'0, trim_to=0'0, roll_forward_to=50'34))", 2024-04-26T01:08:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.077275+0000", 2024-04-26T01:08:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 3.9730022200000001, 2024-04-26T01:08:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0068189660000000001, 2024-04-26T01:08:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077275+0000", 2024-04-26T01:08:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077275+0000", 2024-04-26T01:08:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077277+0000", 2024-04-26T01:08:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2400000000000002e-06 2024-04-26T01:08:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077303+0000", 2024-04-26T01:08:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6024e-05 2024-04-26T01:08:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077751+0000", 2024-04-26T01:08:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00044725800000000002 2024-04-26T01:08:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077772+0000", 2024-04-26T01:08:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1503999999999999e-05 2024-04-26T01:08:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.078004+0000", 2024-04-26T01:08:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000231597 2024-04-26T01:08:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.078031+0000", 2024-04-26T01:08:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7300999999999998e-05 2024-04-26T01:08:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.084015+0000", 2024-04-26T01:08:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0059841640000000001 2024-04-26T01:08:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.084094+0000", 2024-04-26T01:08:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8877999999999999e-05 2024-04-26T01:08:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/30 ECSubWrite(tid=46, reqid=osd.14.0:45, at_version=0'0, trim_to=0'0, roll_forward_to=50'32))", 2024-04-26T01:08:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.078915+0000", 2024-04-26T01:08:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 3.971362659, 2024-04-26T01:08:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0064715049999999998, 2024-04-26T01:08:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.078915+0000", 2024-04-26T01:08:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.078915+0000", 2024-04-26T01:08:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.078916+0000", 2024-04-26T01:08:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.716e-06 2024-04-26T01:08:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.078945+0000", 2024-04-26T01:08:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8286999999999999e-05 2024-04-26T01:08:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.079034+0000", 2024-04-26T01:08:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.9517000000000004e-05 2024-04-26T01:08:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.079056+0000", 2024-04-26T01:08:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1886999999999999e-05 2024-04-26T01:08:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.079939+0000", 2024-04-26T01:08:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00088330099999999996 2024-04-26T01:08:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.079965+0000", 2024-04-26T01:08:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5525999999999999e-05 2024-04-26T01:08:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.085289+0000", 2024-04-26T01:08:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.005324067 2024-04-26T01:08:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.085386+0000", 2024-04-26T01:08:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7204e-05 2024-04-26T01:08:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.149 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:20.150 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-26T01:08:20.151 INFO:tasks.rados.rados.0.smithi164.stdout:208: done (1 left) 2024-04-26T01:08:20.151 INFO:tasks.rados.rados.0.smithi164.stdout:209: done (0 left) 2024-04-26T01:08:20.151 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 35 current snap is 13 2024-04-26T01:08:20.151 INFO:tasks.rados.rados.0.smithi164.stdout:210: seq_num 105 ranges {1359872=606208,1966080=688128,2654208=475136} 2024-04-26T01:08:20.152 INFO:tasks.rados.rados.0.smithi164.stdout:210: writing smithi16428812-35 from 1359872 to 1966080 tid 1 2024-04-26T01:08:20.152 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:08:20.153 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:20.145+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:20.156 INFO:tasks.rados.rados.0.smithi164.stdout:210: writing smithi16428812-35 from 1966080 to 2654208 tid 2 2024-04-26T01:08:20.163 INFO:tasks.rados.rados.0.smithi164.stdout:210: writing smithi16428812-35 from 2654208 to 3129344 tid 3 2024-04-26T01:08:20.163 INFO:tasks.rados.rados.0.smithi164.stdout:211: snap_remove snap 10 2024-04-26T01:08:20.214 INFO:teuthology.orchestra.run.smithi164.stdout:fsck success 2024-04-26T01:08:20.226 INFO:tasks.thrashosds.thrasher:resharding successfully completed 2024-04-26T01:08:20.227 INFO:tasks.thrashosds.thrasher:Reviving osd 11 2024-04-26T01:08:20.227 INFO:tasks.ceph.osd.11:Restarting daemon 2024-04-26T01:08:20.227 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-04-26T01:08:20.252 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:20.245+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:20.271 INFO:tasks.ceph.osd.11:Started 2024-04-26T01:08:20.271 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:08:20.330 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:20.330 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:20.330 INFO:teuthology.orchestra.run.smithi012.stdout: "num_ops": 0 2024-04-26T01:08:20.331 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:20.345 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:08:20.352 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:20.352+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:20.387 INFO:teuthology.orchestra.run.smithi164.stderr:no valid command found; 10 closest matches: 2024-04-26T01:08:20.387 INFO:teuthology.orchestra.run.smithi164.stderr:0 2024-04-26T01:08:20.387 INFO:teuthology.orchestra.run.smithi164.stderr:1 2024-04-26T01:08:20.387 INFO:teuthology.orchestra.run.smithi164.stderr:2 2024-04-26T01:08:20.387 INFO:teuthology.orchestra.run.smithi164.stderr:abort 2024-04-26T01:08:20.388 INFO:teuthology.orchestra.run.smithi164.stderr:assert 2024-04-26T01:08:20.388 INFO:teuthology.orchestra.run.smithi164.stderr:config diff 2024-04-26T01:08:20.388 INFO:teuthology.orchestra.run.smithi164.stderr:config diff get 2024-04-26T01:08:20.388 INFO:teuthology.orchestra.run.smithi164.stderr:config get 2024-04-26T01:08:20.388 INFO:teuthology.orchestra.run.smithi164.stderr:config help [] 2024-04-26T01:08:20.388 INFO:teuthology.orchestra.run.smithi164.stderr:config set ... 2024-04-26T01:08:20.388 INFO:teuthology.orchestra.run.smithi164.stderr:admin_socket: invalid command 2024-04-26T01:08:20.390 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-26T01:08:20.391 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-11, ['dump_ops_in_flight'] 2024-04-26T01:08:20.453 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:20.452+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:20.518 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:20.518 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:20.518 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:08:20.519 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:08:20.519 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:20.532 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:08:20.577 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:20.573+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:20.654 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:08:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:08:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:08:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/30 ECSubWrite(tid=9, reqid=client.4521.0:38, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.327957+0000", 2024-04-26T01:08:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.323965059999999, 2024-04-26T01:08:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.029348438000000001, 2024-04-26T01:08:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327957+0000", 2024-04-26T01:08:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327957+0000", 2024-04-26T01:08:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327958+0000", 2024-04-26T01:08:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.6300000000000005e-07 2024-04-26T01:08:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327988+0000", 2024-04-26T01:08:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.0577000000000002e-05 2024-04-26T01:08:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.354381+0000", 2024-04-26T01:08:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.026392280000000001 2024-04-26T01:08:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.354407+0000", 2024-04-26T01:08:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6783000000000001e-05 2024-04-26T01:08:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.354542+0000", 2024-04-26T01:08:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013464299999999999 2024-04-26T01:08:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.354563+0000", 2024-04-26T01:08:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0744999999999999e-05 2024-04-26T01:08:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357241+0000", 2024-04-26T01:08:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0026777929999999999 2024-04-26T01:08:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357306+0000", 2024-04-26T01:08:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.5054000000000004e-05 2024-04-26T01:08:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/30 ECSubWrite(tid=16, reqid=client.4521.0:65, at_version=33'8, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:08:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.367444+0000", 2024-04-26T01:08:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.284478599, 2024-04-26T01:08:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.014051770999999999, 2024-04-26T01:08:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.367444+0000", 2024-04-26T01:08:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.367444+0000", 2024-04-26T01:08:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.367445+0000", 2024-04-26T01:08:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5460000000000001e-06 2024-04-26T01:08:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.367475+0000", 2024-04-26T01:08:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9869000000000002e-05 2024-04-26T01:08:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.378932+0000", 2024-04-26T01:08:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011457119999999999 2024-04-26T01:08:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.378944+0000", 2024-04-26T01:08:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.182e-05 2024-04-26T01:08:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.379096+0000", 2024-04-26T01:08:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000151743 2024-04-26T01:08:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.379122+0000", 2024-04-26T01:08:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6563e-05 2024-04-26T01:08:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.381451+0000", 2024-04-26T01:08:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0023285419999999999 2024-04-26T01:08:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.381495+0000", 2024-04-26T01:08:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.4567999999999997e-05 2024-04-26T01:08:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpRead(3.1s2 33/30 ECSubRead(tid=23, to_read={3:8f91d505:::smithi16428812-26:head=0,4096,0}, subchunks={3:8f91d505:::smithi16428812-26:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.390419+0000", 2024-04-26T01:08:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.261503355999999, 2024-04-26T01:08:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013787013000000001, 2024-04-26T01:08:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.390419+0000", 2024-04-26T01:08:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.390419+0000", 2024-04-26T01:08:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.390420+0000", 2024-04-26T01:08:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6899999999999996e-07 2024-04-26T01:08:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.390429+0000", 2024-04-26T01:08:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.4010000000000003e-06 2024-04-26T01:08:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.403725+0000", 2024-04-26T01:08:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013295655999999999 2024-04-26T01:08:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.403737+0000", 2024-04-26T01:08:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2034e-05 2024-04-26T01:08:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.403850+0000", 2024-04-26T01:08:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000112596 2024-04-26T01:08:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.404206+0000", 2024-04-26T01:08:20.678 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00035635700000000002 2024-04-26T01:08:20.678 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.678 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.678 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpRead(3.1s2 33/30 ECSubRead(tid=30, to_read={3:88871701:::smithi16428812-34:head=0,4096,0}, subchunks={3:88871701:::smithi16428812-34:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.420990+0000", 2024-04-26T01:08:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.230932405000001, 2024-04-26T01:08:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.028504367999999999, 2024-04-26T01:08:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.420990+0000", 2024-04-26T01:08:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.420990+0000", 2024-04-26T01:08:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.420991+0000", 2024-04-26T01:08:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.254e-06 2024-04-26T01:08:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.421000+0000", 2024-04-26T01:08:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.8370000000000003e-06 2024-04-26T01:08:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.448556+0000", 2024-04-26T01:08:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.027556243000000001 2024-04-26T01:08:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.448583+0000", 2024-04-26T01:08:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6761e-05 2024-04-26T01:08:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.449059+0000", 2024-04-26T01:08:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00047558499999999999 2024-04-26T01:08:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.449494+0000", 2024-04-26T01:08:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00043568799999999999 2024-04-26T01:08:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:487 3.cs0 3:3bb987cc:::smithi16428812-47:head [read 0~1884160,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-04-26T01:08:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.037904+0000", 2024-04-26T01:08:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.614018325, 2024-04-26T01:08:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.044199702, 2024-04-26T01:08:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 487 2024-04-26T01:08:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037904+0000", 2024-04-26T01:08:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037904+0000", 2024-04-26T01:08:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037908+0000", 2024-04-26T01:08:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.1829999999999996e-06 2024-04-26T01:08:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037926+0000", 2024-04-26T01:08:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7859999999999998e-05 2024-04-26T01:08:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.694 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037974+0000", 2024-04-26T01:08:20.694 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.8494e-05 2024-04-26T01:08:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037999+0000", 2024-04-26T01:08:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4899999999999999e-05 2024-04-26T01:08:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038165+0000", 2024-04-26T01:08:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016607699999999999 2024-04-26T01:08:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op must wait for map", 2024-04-26T01:08:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038179+0000", 2024-04-26T01:08:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.403e-05 2024-04-26T01:08:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.049988+0000", 2024-04-26T01:08:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011808206999999999 2024-04-26T01:08:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.050141+0000", 2024-04-26T01:08:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015328799999999999 2024-04-26T01:08:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.082104+0000", 2024-04-26T01:08:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.031962663000000002 2024-04-26T01:08:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:488 3.cs0 3:3bb987cc:::smithi16428812-47:head [read 0~1884160,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-04-26T01:08:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.038063+0000", 2024-04-26T01:08:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.6138597609999996, 2024-04-26T01:08:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.047336889, 2024-04-26T01:08:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 488 2024-04-26T01:08:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038063+0000", 2024-04-26T01:08:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038063+0000", 2024-04-26T01:08:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038064+0000", 2024-04-26T01:08:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3799999999999999e-06 2024-04-26T01:08:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038075+0000", 2024-04-26T01:08:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0993e-05 2024-04-26T01:08:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038096+0000", 2024-04-26T01:08:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1284000000000002e-05 2024-04-26T01:08:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038112+0000", 2024-04-26T01:08:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5577e-05 2024-04-26T01:08:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038366+0000", 2024-04-26T01:08:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025445099999999999 2024-04-26T01:08:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038390+0000", 2024-04-26T01:08:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3782999999999999e-05 2024-04-26T01:08:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051105+0000", 2024-04-26T01:08:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012714757 2024-04-26T01:08:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051362+0000", 2024-04-26T01:08:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025746800000000001 2024-04-26T01:08:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.085399+0000", 2024-04-26T01:08:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.034037195999999999 2024-04-26T01:08:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:489 3.cs0 3:3bb987cc:::smithi16428812-47:head [read 0~1884160,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-04-26T01:08:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.038135+0000", 2024-04-26T01:08:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.613787651, 2024-04-26T01:08:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.049787337000000001, 2024-04-26T01:08:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 489 2024-04-26T01:08:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038135+0000", 2024-04-26T01:08:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038135+0000", 2024-04-26T01:08:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038136+0000", 2024-04-26T01:08:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1739999999999999e-06 2024-04-26T01:08:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038144+0000", 2024-04-26T01:08:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.746e-06 2024-04-26T01:08:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038167+0000", 2024-04-26T01:08:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2977e-05 2024-04-26T01:08:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038182+0000", 2024-04-26T01:08:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5543e-05 2024-04-26T01:08:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038497+0000", 2024-04-26T01:08:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00031515200000000002 2024-04-26T01:08:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038521+0000", 2024-04-26T01:08:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3706999999999999e-05 2024-04-26T01:08:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052089+0000", 2024-04-26T01:08:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013568347 2024-04-26T01:08:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052357+0000", 2024-04-26T01:08:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00026783200000000002 2024-04-26T01:08:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.087922+0000", 2024-04-26T01:08:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.035564858999999997 2024-04-26T01:08:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpRead(3.cs0 50/44 ECSubRead(tid=19, to_read={3:3bb987cc:::smithi16428812-47:head=0,471040,0}, subchunks={3:3bb987cc:::smithi16428812-47:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.050864+0000", 2024-04-26T01:08:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.601058643, 2024-04-26T01:08:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.023549771000000001, 2024-04-26T01:08:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.050864+0000", 2024-04-26T01:08:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.050864+0000", 2024-04-26T01:08:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.050864+0000", 2024-04-26T01:08:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.050864+0000", 2024-04-26T01:08:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053143+0000", 2024-04-26T01:08:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0022794400000000002 2024-04-26T01:08:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053173+0000", 2024-04-26T01:08:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9635999999999999e-05 2024-04-26T01:08:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.069570+0000", 2024-04-26T01:08:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.016397047000000001 2024-04-26T01:08:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.074413+0000", 2024-04-26T01:08:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0048436479999999999 2024-04-26T01:08:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpRead(3.cs0 50/44 ECSubRead(tid=20, to_read={3:3bb987cc:::smithi16428812-47:head=0,471040,0}, subchunks={3:3bb987cc:::smithi16428812-47:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.051911+0000", 2024-04-26T01:08:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.6000107950000002, 2024-04-26T01:08:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.023435303000000001, 2024-04-26T01:08:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051911+0000", 2024-04-26T01:08:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051911+0000", 2024-04-26T01:08:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051911+0000", 2024-04-26T01:08:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051911+0000", 2024-04-26T01:08:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053257+0000", 2024-04-26T01:08:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001345759 2024-04-26T01:08:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053301+0000", 2024-04-26T01:08:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3482999999999997e-05 2024-04-26T01:08:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075035+0000", 2024-04-26T01:08:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.021734690000000001 2024-04-26T01:08:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075347+0000", 2024-04-26T01:08:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00031137100000000001 2024-04-26T01:08:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpRead(3.cs0 50/44 ECSubRead(tid=21, to_read={3:3bb987cc:::smithi16428812-47:head=0,471040,0}, subchunks={3:3bb987cc:::smithi16428812-47:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.052958+0000", 2024-04-26T01:08:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.598964294, 2024-04-26T01:08:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.023405313000000001, 2024-04-26T01:08:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052958+0000", 2024-04-26T01:08:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052958+0000", 2024-04-26T01:08:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052958+0000", 2024-04-26T01:08:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052958+0000", 2024-04-26T01:08:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053516+0000", 2024-04-26T01:08:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00055782599999999998 2024-04-26T01:08:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053561+0000", 2024-04-26T01:08:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.4777000000000002e-05 2024-04-26T01:08:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075975+0000", 2024-04-26T01:08:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.022413968999999999 2024-04-26T01:08:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076363+0000", 2024-04-26T01:08:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00038874100000000001 2024-04-26T01:08:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=19, attrs_read=0))", 2024-04-26T01:08:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.053888+0000", 2024-04-26T01:08:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.5980339670000001, 2024-04-26T01:08:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.021805537, 2024-04-26T01:08:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053888+0000", 2024-04-26T01:08:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053888+0000", 2024-04-26T01:08:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053890+0000", 2024-04-26T01:08:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9870000000000002e-06 2024-04-26T01:08:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055110+0000", 2024-04-26T01:08:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0012193379999999999 2024-04-26T01:08:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055613+0000", 2024-04-26T01:08:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00050335600000000005 2024-04-26T01:08:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055632+0000", 2024-04-26T01:08:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.855e-05 2024-04-26T01:08:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075594+0000", 2024-04-26T01:08:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.019962005000000001 2024-04-26T01:08:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075694+0000", 2024-04-26T01:08:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000100301 2024-04-26T01:08:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:08:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.055120+0000", 2024-04-26T01:08:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.5968027149999999, 2024-04-26T01:08:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.023938912999999999, 2024-04-26T01:08:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055120+0000", 2024-04-26T01:08:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055120+0000", 2024-04-26T01:08:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055121+0000", 2024-04-26T01:08:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.367e-06 2024-04-26T01:08:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055787+0000", 2024-04-26T01:08:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000665702 2024-04-26T01:08:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063149+0000", 2024-04-26T01:08:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0073619590000000004 2024-04-26T01:08:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063177+0000", 2024-04-26T01:08:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8592999999999999e-05 2024-04-26T01:08:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.078938+0000", 2024-04-26T01:08:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.015760687999999998 2024-04-26T01:08:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.079058+0000", 2024-04-26T01:08:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000120604 2024-04-26T01:08:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 48/44 ECSubReadReply(tid=19, attrs_read=0))", 2024-04-26T01:08:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.055536+0000", 2024-04-26T01:08:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.5963859999999999, 2024-04-26T01:08:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.019985012999999999, 2024-04-26T01:08:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055536+0000", 2024-04-26T01:08:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055536+0000", 2024-04-26T01:08:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055538+0000", 2024-04-26T01:08:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5969999999999999e-06 2024-04-26T01:08:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057540+0000", 2024-04-26T01:08:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0020016449999999998 2024-04-26T01:08:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063281+0000", 2024-04-26T01:08:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0057416749999999999 2024-04-26T01:08:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063292+0000", 2024-04-26T01:08:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0913999999999999e-05 2024-04-26T01:08:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075418+0000", 2024-04-26T01:08:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012126296999999999 2024-04-26T01:08:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075521+0000", 2024-04-26T01:08:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000102885 2024-04-26T01:08:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:08:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.055795+0000", 2024-04-26T01:08:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.5961271190000001, 2024-04-26T01:08:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.02721587, 2024-04-26T01:08:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055795+0000", 2024-04-26T01:08:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055795+0000", 2024-04-26T01:08:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055796+0000", 2024-04-26T01:08:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.057e-06 2024-04-26T01:08:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.056601+0000", 2024-04-26T01:08:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00080521399999999995 2024-04-26T01:08:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063232+0000", 2024-04-26T01:08:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0066301579999999997 2024-04-26T01:08:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063246+0000", 2024-04-26T01:08:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4124000000000001e-05 2024-04-26T01:08:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.082918+0000", 2024-04-26T01:08:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.019672486999999999 2024-04-26T01:08:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.083011+0000", 2024-04-26T01:08:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2830000000000004e-05 2024-04-26T01:08:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:08:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.057551+0000", 2024-04-26T01:08:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.5943708680000004, 2024-04-26T01:08:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.019369432999999998, 2024-04-26T01:08:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057551+0000", 2024-04-26T01:08:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057551+0000", 2024-04-26T01:08:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057553+0000", 2024-04-26T01:08:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.144e-06 2024-04-26T01:08:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061928+0000", 2024-04-26T01:08:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0043754420000000002 2024-04-26T01:08:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063323+0000", 2024-04-26T01:08:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001394593 2024-04-26T01:08:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063333+0000", 2024-04-26T01:08:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0200000000000001e-05 2024-04-26T01:08:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076732+0000", 2024-04-26T01:08:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013399576999999999 2024-04-26T01:08:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076921+0000", 2024-04-26T01:08:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00018847699999999999 2024-04-26T01:08:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=19, attrs_read=0))", 2024-04-26T01:08:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.058385+0000", 2024-04-26T01:08:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.5935371570000001, 2024-04-26T01:08:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.017490070999999999, 2024-04-26T01:08:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058385+0000", 2024-04-26T01:08:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058385+0000", 2024-04-26T01:08:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058387+0000", 2024-04-26T01:08:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9429999999999999e-06 2024-04-26T01:08:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060946+0000", 2024-04-26T01:08:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002559024 2024-04-26T01:08:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061804+0000", 2024-04-26T01:08:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00085817 2024-04-26T01:08:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061843+0000", 2024-04-26T01:08:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.8637000000000003e-05 2024-04-26T01:08:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075773+0000", 2024-04-26T01:08:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.01392994 2024-04-26T01:08:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075875+0000", 2024-04-26T01:08:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000102357 2024-04-26T01:08:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:08:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.060960+0000", 2024-04-26T01:08:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.5909621290000002, 2024-04-26T01:08:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.018280428000000001, 2024-04-26T01:08:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060960+0000", 2024-04-26T01:08:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060960+0000", 2024-04-26T01:08:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060961+0000", 2024-04-26T01:08:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.133e-06 2024-04-26T01:08:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065993+0000", 2024-04-26T01:08:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.005031354 2024-04-26T01:08:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.067220+0000", 2024-04-26T01:08:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0012277290000000001 2024-04-26T01:08:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.067239+0000", 2024-04-26T01:08:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8297000000000001e-05 2024-04-26T01:08:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.079129+0000", 2024-04-26T01:08:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011890117 2024-04-26T01:08:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.079241+0000", 2024-04-26T01:08:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000111798 2024-04-26T01:08:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:08:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.061940+0000", 2024-04-26T01:08:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.5899822090000004, 2024-04-26T01:08:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.020587278000000001, 2024-04-26T01:08:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061940+0000", 2024-04-26T01:08:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061940+0000", 2024-04-26T01:08:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061941+0000", 2024-04-26T01:08:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.404e-06 2024-04-26T01:08:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.062953+0000", 2024-04-26T01:08:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0010116999999999999 2024-04-26T01:08:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063362+0000", 2024-04-26T01:08:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000409002 2024-04-26T01:08:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063372+0000", 2024-04-26T01:08:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.026e-05 2024-04-26T01:08:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.082380+0000", 2024-04-26T01:08:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.019007567999999999 2024-04-26T01:08:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.082527+0000", 2024-04-26T01:08:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014734399999999999 2024-04-26T01:08:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:08:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.066015+0000", 2024-04-26T01:08:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.5859068780000003, 2024-04-26T01:08:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.017097062999999999, 2024-04-26T01:08:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066015+0000", 2024-04-26T01:08:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066015+0000", 2024-04-26T01:08:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066017+0000", 2024-04-26T01:08:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.457e-06 2024-04-26T01:08:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066964+0000", 2024-04-26T01:08:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00094692300000000001 2024-04-26T01:08:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.069711+0000", 2024-04-26T01:08:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0027477119999999998 2024-04-26T01:08:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.069731+0000", 2024-04-26T01:08:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9879999999999999e-05 2024-04-26T01:08:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.083047+0000", 2024-04-26T01:08:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013315641 2024-04-26T01:08:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.083112+0000", 2024-04-26T01:08:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.5450000000000005e-05 2024-04-26T01:08:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:08:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.076343+0000", 2024-04-26T01:08:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 4.5755796179999999, 2024-04-26T01:08:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.014954251999999999, 2024-04-26T01:08:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076343+0000", 2024-04-26T01:08:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076343+0000", 2024-04-26T01:08:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076343+0000", 2024-04-26T01:08:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076343+0000", 2024-04-26T01:08:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076393+0000", 2024-04-26T01:08:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0824e-05 2024-04-26T01:08:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076412+0000", 2024-04-26T01:08:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8297000000000001e-05 2024-04-26T01:08:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.085554+0000", 2024-04-26T01:08:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0091421560000000002 2024-04-26T01:08:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.091297+0000", 2024-04-26T01:08:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.005742975 2024-04-26T01:08:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:20.762 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:20.762 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:20.649+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:20.763 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:08:20.808 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:20.805+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:20.866 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:20.864+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:20.875 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:20.876+0000 7ff28a2227c0 -1 Falling back to public interface 2024-04-26T01:08:20.939 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:20.939 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:20.939 INFO:teuthology.orchestra.run.smithi012.stdout: "num_ops": 0 2024-04-26T01:08:20.940 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:20.955 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:08:20.966 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:20.964+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:21.034 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-04-26T01:08:21.039 INFO:tasks.rados.rados.0.smithi164.stdout:211: done (1 left) 2024-04-26T01:08:21.039 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 44 current snap is 13 2024-04-26T01:08:21.039 INFO:tasks.rados.rados.0.smithi164.stdout:210: finishing write tid 1 to smithi16428812-35 2024-04-26T01:08:21.040 INFO:tasks.rados.rados.0.smithi164.stdout:210: finishing write tid 2 to smithi16428812-35 2024-04-26T01:08:21.040 INFO:tasks.rados.rados.0.smithi164.stdout:210: finishing write tid 3 to smithi16428812-35 2024-04-26T01:08:21.040 INFO:tasks.rados.rados.0.smithi164.stdout:210: finishing write tid 4 to smithi16428812-35 2024-04-26T01:08:21.040 INFO:tasks.rados.rados.0.smithi164.stdout:210: finishing write tid 5 to smithi16428812-35 2024-04-26T01:08:21.040 INFO:tasks.rados.rados.0.smithi164.stdout:210: oid 35 updating version 0 to 35 2024-04-26T01:08:21.040 INFO:tasks.rados.rados.0.smithi164.stdout:210: oid 35 updating version 35 to 38 2024-04-26T01:08:21.040 INFO:tasks.rados.rados.0.smithi164.stdout:210: oid 35 version 38 is already newer than 37 2024-04-26T01:08:21.040 INFO:tasks.rados.rados.0.smithi164.stdout:210: oid 35 version 38 is already newer than 36 2024-04-26T01:08:21.040 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 35 v 38 (ObjNum 105 snap 13 seq_num 105) dirty exists 2024-04-26T01:08:21.040 INFO:tasks.rados.rados.0.smithi164.stdout:210: left oid 35 (ObjNum 105 snap 13 seq_num 105) 2024-04-26T01:08:21.040 INFO:tasks.rados.rados.0.smithi164.stdout:212: seq_num 106 ranges {2080768=622592,2703360=442368,3145728=524288,3670016=196608} 2024-04-26T01:08:21.067 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:21.065+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:21.074 INFO:tasks.rados.rados.0.smithi164.stdout:212: writing smithi16428812-44 from 2080768 to 2703360 tid 1 2024-04-26T01:08:21.080 INFO:tasks.rados.rados.0.smithi164.stdout:212: writing smithi16428812-44 from 2703360 to 3145728 tid 2 2024-04-26T01:08:21.088 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:21.088 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:21.088 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:08:21.088 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:08:21.088 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:21.088 INFO:tasks.rados.rados.0.smithi164.stdout:212: writing smithi16428812-44 from 3145728 to 3670016 tid 3 2024-04-26T01:08:21.091 INFO:tasks.rados.rados.0.smithi164.stdout:212: writing smithi16428812-44 from 3670016 to 3866624 tid 4 2024-04-26T01:08:21.091 INFO:tasks.rados.rados.0.smithi164.stdout:210: done (1 left) 2024-04-26T01:08:21.092 INFO:tasks.rados.rados.0.smithi164.stdout:213: copy_from oid 43 from oid 40 current snap is 13 2024-04-26T01:08:21.092 INFO:tasks.rados.rados.0.smithi164.stdout:214: copy_from oid 7 from oid 27 current snap is 13 2024-04-26T01:08:21.092 INFO:tasks.rados.rados.0.smithi164.stdout:215: delete oid 48 current snap is 13 2024-04-26T01:08:21.105 INFO:tasks.rados.rados.0.smithi164.stdout:212: finishing write tid 1 to smithi16428812-44 2024-04-26T01:08:21.113 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:08:21.114 INFO:tasks.rados.rados.0.smithi164.stdout:212: finishing write tid 2 to smithi16428812-44 2024-04-26T01:08:21.116 INFO:tasks.rados.rados.0.smithi164.stdout:212: finishing write tid 3 to smithi16428812-44 2024-04-26T01:08:21.116 INFO:tasks.rados.rados.0.smithi164.stdout:212: finishing write tid 4 to smithi16428812-44 2024-04-26T01:08:21.118 INFO:tasks.rados.rados.0.smithi164.stdout:212: finishing write tid 5 to smithi16428812-44 2024-04-26T01:08:21.119 INFO:tasks.rados.rados.0.smithi164.stdout:215: done (3 left) 2024-04-26T01:08:21.119 INFO:tasks.rados.rados.0.smithi164.stdout:216: snap_remove snap 5 2024-04-26T01:08:21.147 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:08:21.200 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:21.197+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:21.269 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:21.268+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:21.285 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:08:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:08:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:08:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-26T01:08:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:46.050143+0000", 2024-04-26T01:08:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 35.232906642000003, 2024-04-26T01:08:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.36180093800000002, 2024-04-26T01:08:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:08:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:08:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 1 2024-04-26T01:08:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050143+0000", 2024-04-26T01:08:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050143+0000", 2024-04-26T01:08:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050146+0000", 2024-04-26T01:08:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7829999999999999e-06 2024-04-26T01:08:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050160+0000", 2024-04-26T01:08:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4314000000000001e-05 2024-04-26T01:08:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050182+0000", 2024-04-26T01:08:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2407000000000001e-05 2024-04-26T01:08:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050200+0000", 2024-04-26T01:08:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7547e-05 2024-04-26T01:08:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050332+0000", 2024-04-26T01:08:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013237700000000001 2024-04-26T01:08:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for peered", 2024-04-26T01:08:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050361+0000", 2024-04-26T01:08:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8704000000000002e-05 2024-04-26T01:08:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.407786+0000", 2024-04-26T01:08:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.35742484899999999 2024-04-26T01:08:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.408252+0000", 2024-04-26T01:08:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00046585099999999998 2024-04-26T01:08:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:08:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.408586+0000", 2024-04-26T01:08:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033444200000000001 2024-04-26T01:08:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:08:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.409758+0000", 2024-04-26T01:08:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001171718 2024-04-26T01:08:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:08:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.411769+0000", 2024-04-26T01:08:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0020110520000000002 2024-04-26T01:08:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.411870+0000", 2024-04-26T01:08:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000101409 2024-04-26T01:08:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.411944+0000", 2024-04-26T01:08:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.3485000000000005e-05 2024-04-26T01:08:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:90 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-26T01:08:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:46.601265+0000", 2024-04-26T01:08:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 34.68178442, 2024-04-26T01:08:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98632521799999995, 2024-04-26T01:08:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:08:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:08:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 90 2024-04-26T01:08:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601265+0000", 2024-04-26T01:08:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601265+0000", 2024-04-26T01:08:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601267+0000", 2024-04-26T01:08:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.37e-06 2024-04-26T01:08:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601418+0000", 2024-04-26T01:08:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015154400000000001 2024-04-26T01:08:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601436+0000", 2024-04-26T01:08:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7666999999999999e-05 2024-04-26T01:08:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601447+0000", 2024-04-26T01:08:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1406999999999999e-05 2024-04-26T01:08:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.602924+0000", 2024-04-26T01:08:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0014766899999999999 2024-04-26T01:08:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.603071+0000", 2024-04-26T01:08:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014674600000000001 2024-04-26T01:08:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:08:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.603406+0000", 2024-04-26T01:08:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033568500000000001 2024-04-26T01:08:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:08:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.604963+0000", 2024-04-26T01:08:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0015563809999999999 2024-04-26T01:08:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:08:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587422+0000", 2024-04-26T01:08:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98245948699999996 2024-04-26T01:08:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587520+0000", 2024-04-26T01:08:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7650000000000002e-05 2024-04-26T01:08:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587590+0000", 2024-04-26T01:08:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.0591000000000002e-05 2024-04-26T01:08:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_repop_reply(client.4188.0:90 1.0 e24/23 ondisk, result = 0)", 2024-04-26T01:08:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:46.605376+0000", 2024-04-26T01:08:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 34.677674019999998, 2024-04-26T01:08:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98225920899999997, 2024-04-26T01:08:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605376+0000", 2024-04-26T01:08:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605376+0000", 2024-04-26T01:08:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605377+0000", 2024-04-26T01:08:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.407e-06 2024-04-26T01:08:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605384+0000", 2024-04-26T01:08:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.5299999999999999e-06 2024-04-26T01:08:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587134+0000", 2024-04-26T01:08:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98174979699999998 2024-04-26T01:08:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587156+0000", 2024-04-26T01:08:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1704000000000001e-05 2024-04-26T01:08:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587391+0000", 2024-04-26T01:08:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000235052 2024-04-26T01:08:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587412+0000", 2024-04-26T01:08:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0630999999999999e-05 2024-04-26T01:08:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587635+0000", 2024-04-26T01:08:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000223088 2024-04-26T01:08:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:145 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-26T01:08:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:47.981007+0000", 2024-04-26T01:08:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 33.302042157999999, 2024-04-26T01:08:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.057997620999999999, 2024-04-26T01:08:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:08:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:08:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 145 2024-04-26T01:08:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981007+0000", 2024-04-26T01:08:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981007+0000", 2024-04-26T01:08:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981009+0000", 2024-04-26T01:08:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.204e-06 2024-04-26T01:08:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981146+0000", 2024-04-26T01:08:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013786000000000001 2024-04-26T01:08:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981165+0000", 2024-04-26T01:08:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8542999999999998e-05 2024-04-26T01:08:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981177+0000", 2024-04-26T01:08:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1787000000000001e-05 2024-04-26T01:08:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.982190+0000", 2024-04-26T01:08:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0010130180000000001 2024-04-26T01:08:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.982236+0000", 2024-04-26T01:08:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.6445e-05 2024-04-26T01:08:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:08:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.982311+0000", 2024-04-26T01:08:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.4657e-05 2024-04-26T01:08:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:08:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.983568+0000", 2024-04-26T01:08:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001257172 2024-04-26T01:08:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:08:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.038854+0000", 2024-04-26T01:08:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.055285918000000003 2024-04-26T01:08:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.038941+0000", 2024-04-26T01:08:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.6971999999999993e-05 2024-04-26T01:08:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.039005+0000", 2024-04-26T01:08:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.4045000000000001e-05 2024-04-26T01:08:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:15 3.2s0 3:49548768:::smithi16428812-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.304518+0000", 2024-04-26T01:08:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.978531282999999, 2024-04-26T01:08:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.037662963000000001, 2024-04-26T01:08:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 15 2024-04-26T01:08:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304518+0000", 2024-04-26T01:08:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304518+0000", 2024-04-26T01:08:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304520+0000", 2024-04-26T01:08:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.296e-06 2024-04-26T01:08:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304527+0000", 2024-04-26T01:08:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.25e-06 2024-04-26T01:08:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304547+0000", 2024-04-26T01:08:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0307999999999999e-05 2024-04-26T01:08:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304564+0000", 2024-04-26T01:08:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6529999999999999e-05 2024-04-26T01:08:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313033+0000", 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0084690919999999992 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313289+0000", 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025617400000000002 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.334874+0000", 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.021585066999999999 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.335136+0000", 2024-04-26T01:08:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00026158000000000003 2024-04-26T01:08:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.342181+0000", 2024-04-26T01:08:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0070456659999999999 2024-04-26T01:08:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:33 3.2s0 3:410d186f:::smithi16428812-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.328026+0000", 2024-04-26T01:08:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.955023811, 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.065568274999999995, 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 33 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328026+0000", 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328026+0000", 2024-04-26T01:08:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328028+0000", 2024-04-26T01:08:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.357e-06 2024-04-26T01:08:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328040+0000", 2024-04-26T01:08:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2257e-05 2024-04-26T01:08:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328089+0000", 2024-04-26T01:08:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.8164000000000002e-05 2024-04-26T01:08:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328112+0000", 2024-04-26T01:08:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3444000000000001e-05 2024-04-26T01:08:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328852+0000", 2024-04-26T01:08:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00074019899999999998 2024-04-26T01:08:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328981+0000", 2024-04-26T01:08:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012923500000000001 2024-04-26T01:08:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.339980+0000", 2024-04-26T01:08:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010998232 2024-04-26T01:08:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.340144+0000", 2024-04-26T01:08:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016476699999999999 2024-04-26T01:08:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.393594+0000", 2024-04-26T01:08:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.053449620000000003 2024-04-26T01:08:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 33/30 ECSubWriteReply(tid=162, last_complete=33'4, committed=1, applied=1))", 2024-04-26T01:08:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.341290+0000", 2024-04-26T01:08:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.941760041999999, 2024-04-26T01:08:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.050669963999999998, 2024-04-26T01:08:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.341290+0000", 2024-04-26T01:08:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.341290+0000", 2024-04-26T01:08:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.341291+0000", 2024-04-26T01:08:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.237e-06 2024-04-26T01:08:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.341301+0000", 2024-04-26T01:08:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0266e-05 2024-04-26T01:08:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.388586+0000", 2024-04-26T01:08:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.047284633 2024-04-26T01:08:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.388599+0000", 2024-04-26T01:08:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3335e-05 2024-04-26T01:08:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.391887+0000", 2024-04-26T01:08:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0032881400000000002 2024-04-26T01:08:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.391959+0000", 2024-04-26T01:08:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.2353000000000003e-05 2024-04-26T01:08:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:79 3.2s0 3:4b558697:::smithi16428812-27:head [append 0~442368,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.384757+0000", 2024-04-26T01:08:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.898292949000002, 2024-04-26T01:08:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.050291098999999999, 2024-04-26T01:08:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 79 2024-04-26T01:08:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.384757+0000", 2024-04-26T01:08:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.384757+0000", 2024-04-26T01:08:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.384759+0000", 2024-04-26T01:08:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.553e-06 2024-04-26T01:08:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385233+0000", 2024-04-26T01:08:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00047340000000000001 2024-04-26T01:08:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385261+0000", 2024-04-26T01:08:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8405999999999999e-05 2024-04-26T01:08:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385272+0000", 2024-04-26T01:08:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0951000000000001e-05 2024-04-26T01:08:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385543+0000", 2024-04-26T01:08:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00027136800000000002 2024-04-26T01:08:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385733+0000", 2024-04-26T01:08:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00018952800000000001 2024-04-26T01:08:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.387474+0000", 2024-04-26T01:08:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001741156 2024-04-26T01:08:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.392886+0000", 2024-04-26T01:08:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0054121239999999999 2024-04-26T01:08:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.434904+0000", 2024-04-26T01:08:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.042017911999999998 2024-04-26T01:08:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.435048+0000", 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014370100000000001 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:80 3.2s0 3:4b558697:::smithi16428812-27:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.385307+0000", 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.897742484999998, 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.050590553000000003, 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 80 2024-04-26T01:08:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385307+0000", 2024-04-26T01:08:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385307+0000", 2024-04-26T01:08:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385308+0000", 2024-04-26T01:08:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2099999999999995e-07 2024-04-26T01:08:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385315+0000", 2024-04-26T01:08:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.4580000000000004e-06 2024-04-26T01:08:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385330+0000", 2024-04-26T01:08:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4188e-05 2024-04-26T01:08:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385338+0000", 2024-04-26T01:08:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.2190000000000004e-06 2024-04-26T01:08:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.390117+0000", 2024-04-26T01:08:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0047786770000000003 2024-04-26T01:08:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.390334+0000", 2024-04-26T01:08:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00021722000000000001 2024-04-26T01:08:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.391426+0000", 2024-04-26T01:08:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0010922429999999999 2024-04-26T01:08:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.393633+0000", 2024-04-26T01:08:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0022069139999999999 2024-04-26T01:08:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.435677+0000", 2024-04-26T01:08:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.042043589999999999 2024-04-26T01:08:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.435898+0000", 2024-04-26T01:08:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022112299999999999 2024-04-26T01:08:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:81 3.2s0 3:4b558697:::smithi16428812-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.385353+0000", 2024-04-26T01:08:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.897696523, 2024-04-26T01:08:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.12263547800000001, 2024-04-26T01:08:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 81 2024-04-26T01:08:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385353+0000", 2024-04-26T01:08:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385353+0000", 2024-04-26T01:08:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385354+0000", 2024-04-26T01:08:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.6300000000000004e-07 2024-04-26T01:08:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385357+0000", 2024-04-26T01:08:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.4989999999999999e-06 2024-04-26T01:08:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385381+0000", 2024-04-26T01:08:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3643999999999999e-05 2024-04-26T01:08:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385390+0000", 2024-04-26T01:08:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.9090000000000006e-06 2024-04-26T01:08:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.392442+0000", 2024-04-26T01:08:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.007052455 2024-04-26T01:08:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.392642+0000", 2024-04-26T01:08:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000199662 2024-04-26T01:08:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.436978+0000", 2024-04-26T01:08:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.044336042999999999 2024-04-26T01:08:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.437074+0000", 2024-04-26T01:08:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5700999999999997e-05 2024-04-26T01:08:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.507989+0000", 2024-04-26T01:08:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.070914701999999996 2024-04-26T01:08:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 33/30 ECSubWriteReply(tid=164, last_complete=33'5, committed=1, applied=1))", 2024-04-26T01:08:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.393020+0000", 2024-04-26T01:08:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.890029355999999, 2024-04-26T01:08:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.042042138, 2024-04-26T01:08:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.393020+0000", 2024-04-26T01:08:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.393020+0000", 2024-04-26T01:08:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.393022+0000", 2024-04-26T01:08:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5099999999999999e-06 2024-04-26T01:08:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.393033+0000", 2024-04-26T01:08:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1430000000000001e-05 2024-04-26T01:08:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.434612+0000", 2024-04-26T01:08:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.041578801999999998 2024-04-26T01:08:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.434628+0000", 2024-04-26T01:08:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6524999999999998e-05 2024-04-26T01:08:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.434756+0000", 2024-04-26T01:08:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012755199999999999 2024-04-26T01:08:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.435062+0000", 2024-04-26T01:08:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00030631899999999999 2024-04-26T01:08:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.2s0 33/30 ECSubReadReply(tid=169, attrs_read=0))", 2024-04-26T01:08:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.443824+0000", 2024-04-26T01:08:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.839225462999998, 2024-04-26T01:08:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.064178733000000002, 2024-04-26T01:08:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.443824+0000", 2024-04-26T01:08:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.443824+0000", 2024-04-26T01:08:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.443825+0000", 2024-04-26T01:08:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.266e-06 2024-04-26T01:08:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.443845+0000", 2024-04-26T01:08:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9236e-05 2024-04-26T01:08:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.505240+0000", 2024-04-26T01:08:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.061395523 2024-04-26T01:08:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.505256+0000", 2024-04-26T01:08:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6101999999999999e-05 2024-04-26T01:08:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.507809+0000", 2024-04-26T01:08:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0025529820000000001 2024-04-26T01:08:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508003+0000", 2024-04-26T01:08:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019362399999999999 2024-04-26T01:08:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:166 3.2s0 3:410d186f:::smithi16428812-11:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.504084+0000", 2024-04-26T01:08:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.778965358000001, 2024-04-26T01:08:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.062718093000000003, 2024-04-26T01:08:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 166 2024-04-26T01:08:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504084+0000", 2024-04-26T01:08:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504084+0000", 2024-04-26T01:08:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504086+0000", 2024-04-26T01:08:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8819999999999999e-06 2024-04-26T01:08:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504689+0000", 2024-04-26T01:08:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000602634 2024-04-26T01:08:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504716+0000", 2024-04-26T01:08:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6995000000000002e-05 2024-04-26T01:08:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504732+0000", 2024-04-26T01:08:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5896000000000002e-05 2024-04-26T01:08:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504855+0000", 2024-04-26T01:08:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012368000000000001 2024-04-26T01:08:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.505004+0000", 2024-04-26T01:08:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000148515 2024-04-26T01:08:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.506133+0000", 2024-04-26T01:08:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001129625 2024-04-26T01:08:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508806+0000", 2024-04-26T01:08:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0026721760000000001 2024-04-26T01:08:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566633+0000", 2024-04-26T01:08:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.057827207999999998 2024-04-26T01:08:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566802+0000", 2024-04-26T01:08:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016948199999999999 2024-04-26T01:08:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:167 3.2s0 3:410d186f:::smithi16428812-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.504769+0000", 2024-04-26T01:08:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.778280915, 2024-04-26T01:08:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.18953119900000001, 2024-04-26T01:08:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 167 2024-04-26T01:08:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504769+0000", 2024-04-26T01:08:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504769+0000", 2024-04-26T01:08:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504770+0000", 2024-04-26T01:08:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.356e-06 2024-04-26T01:08:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504777+0000", 2024-04-26T01:08:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.8140000000000003e-06 2024-04-26T01:08:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504791+0000", 2024-04-26T01:08:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4267000000000001e-05 2024-04-26T01:08:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504807+0000", 2024-04-26T01:08:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6030000000000001e-05 2024-04-26T01:08:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.506957+0000", 2024-04-26T01:08:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0021496549999999999 2024-04-26T01:08:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.507039+0000", 2024-04-26T01:08:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1959000000000006e-05 2024-04-26T01:08:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.507462+0000", 2024-04-26T01:08:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00042285800000000002 2024-04-26T01:08:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.511679+0000", 2024-04-26T01:08:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0042174209999999998 2024-04-26T01:08:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694149+0000", 2024-04-26T01:08:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.18246974999999999 2024-04-26T01:08:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694300+0000", 2024-04-26T01:08:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000151089 2024-04-26T01:08:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:168 3.2s0 3:410d186f:::smithi16428812-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.504832+0000", 2024-04-26T01:08:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.778217464000001, 2024-04-26T01:08:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.19389620699999999, 2024-04-26T01:08:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 168 2024-04-26T01:08:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504832+0000", 2024-04-26T01:08:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504832+0000", 2024-04-26T01:08:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504833+0000", 2024-04-26T01:08:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5199999999999995e-07 2024-04-26T01:08:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504838+0000", 2024-04-26T01:08:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.1560000000000003e-06 2024-04-26T01:08:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504853+0000", 2024-04-26T01:08:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.469e-05 2024-04-26T01:08:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504865+0000", 2024-04-26T01:08:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1707e-05 2024-04-26T01:08:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508279+0000", 2024-04-26T01:08:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0034144650000000002 2024-04-26T01:08:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508350+0000", 2024-04-26T01:08:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1166000000000003e-05 2024-04-26T01:08:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694397+0000", 2024-04-26T01:08:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.18604716299999999 2024-04-26T01:08:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694544+0000", 2024-04-26T01:08:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014672200000000001 2024-04-26T01:08:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.698728+0000", 2024-04-26T01:08:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0041841860000000003 2024-04-26T01:08:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 33/30 ECSubWriteReply(tid=173, last_complete=33'8, committed=1, applied=1))", 2024-04-26T01:08:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.508639+0000", 2024-04-26T01:08:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.774410584999998, 2024-04-26T01:08:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.058189232, 2024-04-26T01:08:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508639+0000", 2024-04-26T01:08:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508639+0000", 2024-04-26T01:08:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508640+0000", 2024-04-26T01:08:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.9999999999999997e-07 2024-04-26T01:08:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508646+0000", 2024-04-26T01:08:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.0580000000000003e-06 2024-04-26T01:08:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566182+0000", 2024-04-26T01:08:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.057536449000000003 2024-04-26T01:08:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566201+0000", 2024-04-26T01:08:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8904000000000001e-05 2024-04-26T01:08:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566315+0000", 2024-04-26T01:08:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00011435800000000001 2024-04-26T01:08:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566828+0000", 2024-04-26T01:08:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00051276299999999996 2024-04-26T01:08:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:191 3.2s0 3:4b558697:::smithi16428812-27:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.563295+0000", 2024-04-26T01:08:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.719754649999999, 2024-04-26T01:08:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.13397727300000001, 2024-04-26T01:08:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 191 2024-04-26T01:08:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563295+0000", 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563295+0000", 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563297+0000", 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4140000000000001e-06 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563318+0000", 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0211e-05 2024-04-26T01:08:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563342+0000", 2024-04-26T01:08:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4266000000000001e-05 2024-04-26T01:08:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563353+0000", 2024-04-26T01:08:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0886e-05 2024-04-26T01:08:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563479+0000", 2024-04-26T01:08:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000126651 2024-04-26T01:08:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563655+0000", 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000175713 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.564447+0000", 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00079165500000000005 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.565861+0000", 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001414595 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.697131+0000", 2024-04-26T01:08:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.13126945700000001 2024-04-26T01:08:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.697272+0000", 2024-04-26T01:08:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014142500000000001 2024-04-26T01:08:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 33/30 ECSubWriteReply(tid=174, last_complete=33'9, committed=1, applied=1))", 2024-04-26T01:08:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.566047+0000", 2024-04-26T01:08:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.717002559000001, 2024-04-26T01:08:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.128277063, 2024-04-26T01:08:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566047+0000", 2024-04-26T01:08:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566047+0000", 2024-04-26T01:08:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566048+0000", 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.316e-06 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566138+0000", 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.9281000000000001e-05 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.693854+0000", 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.12771605 2024-04-26T01:08:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.693868+0000", 2024-04-26T01:08:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3910000000000001e-05 2024-04-26T01:08:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694017+0000", 2024-04-26T01:08:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000149865 2024-04-26T01:08:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694324+0000", 2024-04-26T01:08:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00030664100000000002 2024-04-26T01:08:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 38/30 ECSubWriteReply(tid=190, last_complete=38'7, committed=1, applied=1))", 2024-04-26T01:08:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.497964+0000", 2024-04-26T01:08:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 16.785085239000001, 2024-04-26T01:08:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.81206265399999999, 2024-04-26T01:08:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.497964+0000", 2024-04-26T01:08:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.497964+0000", 2024-04-26T01:08:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.497966+0000", 2024-04-26T01:08:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.274e-06 2024-04-26T01:08:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.497979+0000", 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3127e-05 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.309733+0000", 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.81175416700000003 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.309746+0000", 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3523000000000001e-05 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.309899+0000", 2024-04-26T01:08:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015288199999999999 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.310027+0000", 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012768100000000001 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.fs3 50/44 ECSubWrite(tid=225, reqid=client.4521.0:494, at_version=50'41, trim_to=0'0, roll_forward_to=50'38))", 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.091583+0000", 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.1914666140000003, 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.14404377500000001, 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.091583+0000", 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.091583+0000", 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.091585+0000", 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.075e-06 2024-04-26T01:08:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.091712+0000", 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012731099999999999 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.233513+0000", 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.14180088399999999 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.233530+0000", 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6390000000000001e-05 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.233655+0000", 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000125341 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.233671+0000", 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6515999999999999e-05 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.235542+0000", 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001870125 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.235627+0000", 2024-04-26T01:08:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.5133e-05 2024-04-26T01:08:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:21.381 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:21.382 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:21.382 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:21.383 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:21.383 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:21.383 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:08:21.384 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:21.384 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:21.385 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:21.380+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:21.390 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:21.391 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:21.418 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:21.418 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:21.450 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:21.450 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:21.479 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:21.479 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:21.483 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:21.480+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:21.512 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:21.512 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:21.514 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:21.514 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:21.529 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:08:21.543 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:21.543 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:21.578 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:21.578 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:21.584 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:21.581+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:21.611 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:21.611 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:21.612 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-26T01:08:21.646 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:21.646 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:21.648 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:21.648 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:21.662 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:08:21.675 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:21.675 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:21.708 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:21.708 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:21.708 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:21.708+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:21.739 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:21.736+0000 7fe1d577b640 0 -- 172.21.15.8:0/2938339492 >> [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] conn(0x7fe1d0223be0 msgr2=0x7fe1d0225fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:08:21.748 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:21.754 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T01:08:21.755 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T01:08:21.782 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:21.783 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:21.786 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:21.781+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:21.797 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:08:21.887 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:21.881+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:21.966 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:21.967 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:21.980 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:08:21.992 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:21.989+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:22.041 INFO:tasks.rados.rados.0.smithi164.stdout:216: done (3 left) 2024-04-26T01:08:22.041 INFO:tasks.rados.rados.0.smithi164.stdout:217: copy_from oid 4 from oid 21 current snap is 13 2024-04-26T01:08:22.041 INFO:tasks.rados.rados.0.smithi164.stdout:214: finishing copy_from racing read to smithi16428812-7 2024-04-26T01:08:22.042 INFO:tasks.rados.rados.0.smithi164.stdout:214: finishing copy_from to smithi16428812-7 2024-04-26T01:08:22.042 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 7 v 22 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-26T01:08:22.042 INFO:tasks.rados.rados.0.smithi164.stdout:213: finishing copy_from racing read to smithi16428812-43 2024-04-26T01:08:22.042 INFO:tasks.rados.rados.0.smithi164.stdout:213: finishing copy_from to smithi16428812-43 2024-04-26T01:08:22.042 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 43 v 26 (ObjNum 104 snap 11 seq_num 104) dirty exists 2024-04-26T01:08:22.042 INFO:tasks.rados.rados.0.smithi164.stdout:212: finishing write tid 6 to smithi16428812-44 2024-04-26T01:08:22.042 INFO:tasks.rados.rados.0.smithi164.stdout:212: oid 44 updating version 0 to 31 2024-04-26T01:08:22.042 INFO:tasks.rados.rados.0.smithi164.stdout:212: oid 44 version 31 is already newer than 29 2024-04-26T01:08:22.042 INFO:tasks.rados.rados.0.smithi164.stdout:212: oid 44 updating version 31 to 32 2024-04-26T01:08:22.042 INFO:tasks.rados.rados.0.smithi164.stdout:212: oid 44 version 32 is already newer than 28 2024-04-26T01:08:22.042 INFO:tasks.rados.rados.0.smithi164.stdout:212: oid 44 version 32 is already newer than 30 2024-04-26T01:08:22.042 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 44 v 32 (ObjNum 106 snap 13 seq_num 106) dirty exists 2024-04-26T01:08:22.042 INFO:tasks.rados.rados.0.smithi164.stdout:212: left oid 44 (ObjNum 106 snap 13 seq_num 106) 2024-04-26T01:08:22.042 INFO:tasks.rados.rados.0.smithi164.stdout:212: done (3 left) 2024-04-26T01:08:22.043 INFO:tasks.rados.rados.0.smithi164.stdout:213: done (2 left) 2024-04-26T01:08:22.043 INFO:tasks.rados.rados.0.smithi164.stdout:214: done (1 left) 2024-04-26T01:08:22.043 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 49 current snap is 13 2024-04-26T01:08:22.043 INFO:tasks.rados.rados.0.smithi164.stdout:218: seq_num 108 ranges {2457600=655360,3112960=606208,3719168=475136,4194304=81920} 2024-04-26T01:08:22.090 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:22.088+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:22.091 INFO:tasks.rados.rados.0.smithi164.stdout:218: writing smithi16428812-49 from 2457600 to 3112960 tid 1 2024-04-26T01:08:22.098 INFO:tasks.rados.rados.0.smithi164.stdout:218: writing smithi16428812-49 from 3112960 to 3719168 tid 2 2024-04-26T01:08:22.104 INFO:tasks.rados.rados.0.smithi164.stdout:218: writing smithi16428812-49 from 3719168 to 4194304 tid 3 2024-04-26T01:08:22.105 INFO:tasks.rados.rados.0.smithi164.stdout:218: writing smithi16428812-49 from 4194304 to 4276224 tid 4 2024-04-26T01:08:22.105 INFO:tasks.rados.rados.0.smithi164.stdout:219: copy_from oid 35 from oid 44 current snap is 13 2024-04-26T01:08:22.105 INFO:tasks.rados.rados.0.smithi164.stdout:217: finishing copy_from racing read to smithi16428812-4 2024-04-26T01:08:22.105 INFO:tasks.rados.rados.0.smithi164.stdout:220: delete oid 11 current snap is 13 2024-04-26T01:08:22.117 INFO:tasks.rados.rados.0.smithi164.stdout:217: finishing copy_from to smithi16428812-4 2024-04-26T01:08:22.117 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 4 v 36 (ObjNum 88 snap 6 seq_num 88) dirty exists 2024-04-26T01:08:22.117 INFO:tasks.rados.rados.0.smithi164.stdout:218: finishing write tid 1 to smithi16428812-49 2024-04-26T01:08:22.123 INFO:tasks.rados.rados.0.smithi164.stdout:218: finishing write tid 2 to smithi16428812-49 2024-04-26T01:08:22.126 INFO:tasks.rados.rados.0.smithi164.stdout:218: finishing write tid 3 to smithi16428812-49 2024-04-26T01:08:22.126 INFO:tasks.rados.rados.0.smithi164.stdout:218: finishing write tid 4 to smithi16428812-49 2024-04-26T01:08:22.128 INFO:tasks.rados.rados.0.smithi164.stdout:218: finishing write tid 5 to smithi16428812-49 2024-04-26T01:08:22.131 INFO:tasks.rados.rados.0.smithi164.stdout:218: finishing write tid 6 to smithi16428812-49 2024-04-26T01:08:22.131 INFO:tasks.rados.rados.0.smithi164.stdout:218: oid 49 updating version 0 to 60 2024-04-26T01:08:22.131 INFO:tasks.rados.rados.0.smithi164.stdout:218: oid 49 version 60 is already newer than 59 2024-04-26T01:08:22.131 INFO:tasks.rados.rados.0.smithi164.stdout:218: oid 49 updating version 60 to 62 2024-04-26T01:08:22.131 INFO:tasks.rados.rados.0.smithi164.stdout:218: oid 49 version 62 is already newer than 61 2024-04-26T01:08:22.131 INFO:tasks.rados.rados.0.smithi164.stdout:218: oid 49 version 62 is already newer than 58 2024-04-26T01:08:22.131 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 49 v 62 (ObjNum 108 snap 13 seq_num 108) dirty exists 2024-04-26T01:08:22.131 INFO:tasks.rados.rados.0.smithi164.stdout:218: left oid 49 (ObjNum 108 snap 13 seq_num 108) 2024-04-26T01:08:22.132 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:22.132 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:22.148 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:08:22.189 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:22.184+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:22.265 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:22.265 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:22.279 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:08:22.290 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:22.292+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:22.391 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:22.388+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:22.395 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:22.395 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:22.410 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:08:22.496 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:22.496+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:22.499 INFO:tasks.rados.rados.0.smithi164.stdout:217: done (3 left) 2024-04-26T01:08:22.499 INFO:tasks.rados.rados.0.smithi164.stdout:218: done (2 left) 2024-04-26T01:08:22.499 INFO:tasks.rados.rados.0.smithi164.stdout:220: done (1 left) 2024-04-26T01:08:22.499 INFO:tasks.rados.rados.0.smithi164.stdout:221: read oid 37 snap -1 2024-04-26T01:08:22.499 INFO:tasks.rados.rados.0.smithi164.stdout:221: expect (ObjNum 87 snap 6 seq_num 87) 2024-04-26T01:08:22.500 INFO:tasks.rados.rados.0.smithi164.stdout:222: snap_create 2024-04-26T01:08:22.513 INFO:tasks.rados.rados.0.smithi164.stdout:221: expect (ObjNum 87 snap 6 seq_num 87) 2024-04-26T01:08:22.575 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:22.575 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:22.589 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:08:22.595 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:22.592+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:22.696 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:22.693+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:22.751 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:22.751 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:22.765 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:08:22.795 INFO:tasks.rados.rados.0.smithi164.stdout:219: finishing copy_from racing read to smithi16428812-35 2024-04-26T01:08:22.799 INFO:tasks.rados.rados.0.smithi164.stdout:219: finishing copy_from to smithi16428812-35 2024-04-26T01:08:22.800 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 35 v 32 (ObjNum 106 snap 13 seq_num 106) dirty exists 2024-04-26T01:08:22.812 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:22.812+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:22.890 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:22.890 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:22.898 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:22.896+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:22.904 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:08:22.998 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:22.993+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:23.065 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-26T01:08:23.070 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:23.070 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:23.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:08:23.110 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:23.108+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:23.142 INFO:tasks.rados.rados.0.smithi164.stdout:219: done (2 left) 2024-04-26T01:08:23.143 INFO:tasks.rados.rados.0.smithi164.stdout:221: done (1 left) 2024-04-26T01:08:23.143 INFO:tasks.rados.rados.0.smithi164.stdout:222: done (0 left) 2024-04-26T01:08:23.143 INFO:tasks.rados.rados.0.smithi164.stdout:223: rmattr oid 6 current snap is 14 2024-04-26T01:08:23.143 INFO:tasks.rados.rados.0.smithi164.stdout:223: done (0 left) 2024-04-26T01:08:23.143 INFO:tasks.rados.rados.0.smithi164.stdout:224: copy_from oid 41 from oid 32 current snap is 14 2024-04-26T01:08:23.143 INFO:tasks.rados.rados.0.smithi164.stdout:225: read oid 34 snap -1 2024-04-26T01:08:23.143 INFO:tasks.rados.rados.0.smithi164.stdout:225: expect deleted 2024-04-26T01:08:23.143 INFO:tasks.rados.rados.0.smithi164.stdout:226: rollback oid 9 current snap is 14 2024-04-26T01:08:23.143 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 9 to 13 2024-04-26T01:08:23.143 INFO:tasks.rados.rados.0.smithi164.stdout:227: snap_remove snap 9 2024-04-26T01:08:23.200 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:23.196+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:23.207 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:23.207 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:23.221 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:08:23.301 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:23.296+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:23.350 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:23.350 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:23.364 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:08:23.402 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:23.399+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:23.487 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:23.488 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:23.503 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:08:23.504 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:23.504+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:23.604 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:23.603+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:23.668 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:23.668 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:23.683 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:08:23.704 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:23.703+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:23.756 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:08:23.805 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:23.805+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:23.811 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:23.811 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:23.826 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:08:23.906 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:23.904+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:23.931 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:23.931 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:23.954 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:23.954 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:23.961 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:23.961 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:23.968 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:08:23.993 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:23.993 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:24.014 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:24.016+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:24.021 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:24.022 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:24.052 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:24.052 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:24.084 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:24.085 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:24.089 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:24.090 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:24.103 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:08:24.145 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-04-26T01:08:24.150 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:24.152+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:24.151 INFO:tasks.rados.rados.0.smithi164.stdout:227: done (3 left) 2024-04-26T01:08:24.151 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 19 current snap is 14 2024-04-26T01:08:24.151 INFO:tasks.rados.rados.0.smithi164.stdout:228: seq_num 110 ranges {311296=688128,999424=507904,1507328=655360} 2024-04-26T01:08:24.156 INFO:tasks.rados.rados.0.smithi164.stdout:228: writing smithi16428812-19 from 311296 to 999424 tid 1 2024-04-26T01:08:24.160 INFO:tasks.rados.rados.0.smithi164.stdout:228: writing smithi16428812-19 from 999424 to 1507328 tid 2 2024-04-26T01:08:24.164 INFO:tasks.rados.rados.0.smithi164.stdout:228: writing smithi16428812-19 from 1507328 to 2162688 tid 3 2024-04-26T01:08:24.164 INFO:tasks.rados.rados.0.smithi164.stdout:229: rollback oid 6 current snap is 14 2024-04-26T01:08:24.164 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 6 to 11 2024-04-26T01:08:24.164 INFO:tasks.rados.rados.0.smithi164.stdout:224: finishing copy_from to smithi16428812-41 2024-04-26T01:08:24.164 INFO:tasks.rados.rados.0.smithi164.stdout:224: got expected ENOENT (src dne) 2024-04-26T01:08:24.164 INFO:tasks.rados.rados.0.smithi164.stdout:224: finishing copy_from racing read to smithi16428812-41 2024-04-26T01:08:24.165 INFO:tasks.rados.rados.0.smithi164.stdout:226: finishing rollback tid 0 to smithi16428812-9 2024-04-26T01:08:24.165 INFO:tasks.rados.rados.0.smithi164.stdout:226: finishing rollback tid 1 to smithi16428812-9 2024-04-26T01:08:24.165 INFO:tasks.rados.rados.0.smithi164.stdout:226: finishing rollback tid 2 to smithi16428812-9 2024-04-26T01:08:24.165 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 9 v 47 (ObjNum 99 snap 11 seq_num 99) dirty exists 2024-04-26T01:08:24.165 INFO:tasks.rados.rados.0.smithi164.stdout:224: done (4 left) 2024-04-26T01:08:24.165 INFO:tasks.rados.rados.0.smithi164.stdout:225: done (3 left) 2024-04-26T01:08:24.165 INFO:tasks.rados.rados.0.smithi164.stdout:226: done (2 left) 2024-04-26T01:08:24.165 INFO:tasks.rados.rados.0.smithi164.stdout:230: rmattr oid 47 current snap is 14 2024-04-26T01:08:24.165 INFO:tasks.rados.rados.0.smithi164.stdout:230: done (2 left) 2024-04-26T01:08:24.165 INFO:tasks.rados.rados.0.smithi164.stdout:231: read oid 38 snap -1 2024-04-26T01:08:24.165 INFO:tasks.rados.rados.0.smithi164.stdout:231: expect (ObjNum 75 snap 4 seq_num 75) 2024-04-26T01:08:24.165 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 18 current snap is 14 2024-04-26T01:08:24.165 INFO:tasks.rados.rados.0.smithi164.stdout:232: seq_num 111 ranges {1720320=704512,2424832=655360,3080192=507904} 2024-04-26T01:08:24.178 INFO:tasks.rados.rados.0.smithi164.stdout:232: writing smithi16428812-18 from 1720320 to 2424832 tid 1 2024-04-26T01:08:24.182 INFO:tasks.rados.rados.0.smithi164.stdout:232: writing smithi16428812-18 from 2424832 to 3080192 tid 2 2024-04-26T01:08:24.186 INFO:tasks.rados.rados.0.smithi164.stdout:232: writing smithi16428812-18 from 3080192 to 3588096 tid 3 2024-04-26T01:08:24.186 INFO:tasks.rados.rados.0.smithi164.stdout:233: delete oid 50 current snap is 14 2024-04-26T01:08:24.186 INFO:tasks.rados.rados.0.smithi164.stdout:231: expect (ObjNum 75 snap 4 seq_num 75) 2024-04-26T01:08:24.209 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:24.207+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:24.218 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:24.218 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:24.245 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:24.245 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:24.255 INFO:tasks.rados.rados.0.smithi164.stdout:229: finishing rollback tid 0 to smithi16428812-6 2024-04-26T01:08:24.255 INFO:tasks.rados.rados.0.smithi164.stdout:229: finishing rollback tid 1 to smithi16428812-6 2024-04-26T01:08:24.255 INFO:tasks.rados.rados.0.smithi164.stdout:229: finishing rollback tid 2 to smithi16428812-6 2024-04-26T01:08:24.255 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 6 v 34 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-26T01:08:24.255 INFO:tasks.rados.rados.0.smithi164.stdout:228: finishing write tid 1 to smithi16428812-19 2024-04-26T01:08:24.255 INFO:tasks.rados.rados.0.smithi164.stdout:228: finishing write tid 2 to smithi16428812-19 2024-04-26T01:08:24.255 INFO:tasks.rados.rados.0.smithi164.stdout:228: finishing write tid 3 to smithi16428812-19 2024-04-26T01:08:24.255 INFO:tasks.rados.rados.0.smithi164.stdout:228: finishing write tid 4 to smithi16428812-19 2024-04-26T01:08:24.255 INFO:tasks.rados.rados.0.smithi164.stdout:228: finishing write tid 5 to smithi16428812-19 2024-04-26T01:08:24.256 INFO:tasks.rados.rados.0.smithi164.stdout:228: oid 19 updating version 0 to 53 2024-04-26T01:08:24.256 INFO:tasks.rados.rados.0.smithi164.stdout:228: oid 19 version 53 is already newer than 51 2024-04-26T01:08:24.256 INFO:tasks.rados.rados.0.smithi164.stdout:228: oid 19 updating version 53 to 54 2024-04-26T01:08:24.256 INFO:tasks.rados.rados.0.smithi164.stdout:228: oid 19 version 54 is already newer than 52 2024-04-26T01:08:24.256 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 19 v 54 (ObjNum 110 snap 14 seq_num 110) dirty exists 2024-04-26T01:08:24.256 INFO:tasks.rados.rados.0.smithi164.stdout:228: left oid 19 (ObjNum 110 snap 14 seq_num 110) 2024-04-26T01:08:24.256 INFO:tasks.rados.rados.0.smithi164.stdout:232: finishing write tid 1 to smithi16428812-18 2024-04-26T01:08:24.256 INFO:tasks.rados.rados.0.smithi164.stdout:232: finishing write tid 2 to smithi16428812-18 2024-04-26T01:08:24.256 INFO:tasks.rados.rados.0.smithi164.stdout:232: finishing write tid 3 to smithi16428812-18 2024-04-26T01:08:24.256 INFO:tasks.rados.rados.0.smithi164.stdout:232: finishing write tid 4 to smithi16428812-18 2024-04-26T01:08:24.270 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:24.270 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:24.278 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:24.279 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:24.281 INFO:tasks.rados.rados.0.smithi164.stdout:232: finishing write tid 5 to smithi16428812-18 2024-04-26T01:08:24.282 INFO:tasks.rados.rados.0.smithi164.stdout:232: oid 18 updating version 0 to 70 2024-04-26T01:08:24.282 INFO:tasks.rados.rados.0.smithi164.stdout:232: oid 18 version 70 is already newer than 68 2024-04-26T01:08:24.282 INFO:tasks.rados.rados.0.smithi164.stdout:232: oid 18 updating version 70 to 71 2024-04-26T01:08:24.282 INFO:tasks.rados.rados.0.smithi164.stdout:232: oid 18 version 71 is already newer than 69 2024-04-26T01:08:24.282 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 18 v 71 (ObjNum 111 snap 14 seq_num 111) dirty exists 2024-04-26T01:08:24.282 INFO:tasks.rados.rados.0.smithi164.stdout:232: left oid 18 (ObjNum 111 snap 14 seq_num 111) 2024-04-26T01:08:24.294 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:08:24.306 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:24.306 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:24.338 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:24.338 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:24.339 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:24.340+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:24.339 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-26T01:08:24.341 INFO:tasks.rados.rados.0.smithi164.stdout:228: done (4 left) 2024-04-26T01:08:24.341 INFO:tasks.rados.rados.0.smithi164.stdout:229: done (3 left) 2024-04-26T01:08:24.341 INFO:tasks.rados.rados.0.smithi164.stdout:231: done (2 left) 2024-04-26T01:08:24.341 INFO:tasks.rados.rados.0.smithi164.stdout:232: done (1 left) 2024-04-26T01:08:24.341 INFO:tasks.rados.rados.0.smithi164.stdout:233: done (0 left) 2024-04-26T01:08:24.341 INFO:tasks.rados.rados.0.smithi164.stdout:234: copy_from oid 40 from oid 13 current snap is 14 2024-04-26T01:08:24.342 INFO:tasks.rados.rados.0.smithi164.stdout:235: copy_from oid 38 from oid 48 current snap is 14 2024-04-26T01:08:24.342 INFO:tasks.rados.rados.0.smithi164.stdout:236: read oid 29 snap -1 2024-04-26T01:08:24.342 INFO:tasks.rados.rados.0.smithi164.stdout:236: expect (ObjNum 98 snap 11 seq_num 98) 2024-04-26T01:08:24.342 INFO:tasks.rados.rados.0.smithi164.stdout:237: read oid 49 snap -1 2024-04-26T01:08:24.342 INFO:tasks.rados.rados.0.smithi164.stdout:237: expect (ObjNum 108 snap 13 seq_num 108) 2024-04-26T01:08:24.342 INFO:tasks.rados.rados.0.smithi164.stdout:238: snap_remove snap 12 2024-04-26T01:08:24.380 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:24.380 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:24.410 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:24.410 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:24.410 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:24.405+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:24.441 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:24.441 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:24.448 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:24.444+0000 7f9aa9d9b640 0 -- 172.21.15.8:0/4249013346 >> [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] conn(0x7f9aac2235e0 msgr2=0x7f9aac2259d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:08:24.456 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:24.456 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:24.470 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:08:24.482 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:24.482 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:24.487 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T01:08:24.488 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T01:08:24.511 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:24.505+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:24.595 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:24.596+0000 7ff28a2227c0 -1 osd.11 42 log_to_monitors true 2024-04-26T01:08:24.602 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2024-04-26T01:08:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2024-04-26T01:08:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2024-04-26T01:08:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:28 3.5s0 3:acf4d671:::smithi16428812-10:head [append 0~163840,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321346+0000", 2024-04-26T01:08:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 25.282058736, 2024-04-26T01:08:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015195917, 2024-04-26T01:08:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 28 2024-04-26T01:08:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321346+0000", 2024-04-26T01:08:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321346+0000", 2024-04-26T01:08:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321349+0000", 2024-04-26T01:08:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0879999999999999e-06 2024-04-26T01:08:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321587+0000", 2024-04-26T01:08:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000237698 2024-04-26T01:08:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321610+0000", 2024-04-26T01:08:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2597000000000001e-05 2024-04-26T01:08:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321624+0000", 2024-04-26T01:08:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4365999999999999e-05 2024-04-26T01:08:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321789+0000", 2024-04-26T01:08:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016444499999999999 2024-04-26T01:08:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322181+0000", 2024-04-26T01:08:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00039229 2024-04-26T01:08:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324455+0000", 2024-04-26T01:08:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0022738979999999999 2024-04-26T01:08:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.334069+0000", 2024-04-26T01:08:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.009614793 2024-04-26T01:08:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336369+0000", 2024-04-26T01:08:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0022993689999999999 2024-04-26T01:08:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336542+0000", 2024-04-26T01:08:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017337299999999999 2024-04-26T01:08:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:29 3.5s0 3:acf4d671:::smithi16428812-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321678+0000", 2024-04-26T01:08:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 25.281726689999999, 2024-04-26T01:08:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.016320530999999999, 2024-04-26T01:08:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 29 2024-04-26T01:08:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321678+0000", 2024-04-26T01:08:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321678+0000", 2024-04-26T01:08:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321680+0000", 2024-04-26T01:08:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1680000000000001e-06 2024-04-26T01:08:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321686+0000", 2024-04-26T01:08:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9539999999999998e-06 2024-04-26T01:08:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321702+0000", 2024-04-26T01:08:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5241e-05 2024-04-26T01:08:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321716+0000", 2024-04-26T01:08:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4384e-05 2024-04-26T01:08:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329428+0000", 2024-04-26T01:08:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077114369999999998 2024-04-26T01:08:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329669+0000", 2024-04-26T01:08:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000241561 2024-04-26T01:08:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.331103+0000", 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001433871 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336734+0000", 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.005631306 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337654+0000", 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000919716 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337999+0000", 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034489299999999998 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:30 3.5s0 3:acf4d671:::smithi16428812-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.322117+0000", 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 25.281287634999998, 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.021239319999999999, 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 30 2024-04-26T01:08:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322117+0000", 2024-04-26T01:08:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322117+0000", 2024-04-26T01:08:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322119+0000", 2024-04-26T01:08:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3740000000000001e-06 2024-04-26T01:08:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322127+0000", 2024-04-26T01:08:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.994e-06 2024-04-26T01:08:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322146+0000", 2024-04-26T01:08:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9205999999999999e-05 2024-04-26T01:08:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322159+0000", 2024-04-26T01:08:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3377000000000001e-05 2024-04-26T01:08:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.334766+0000", 2024-04-26T01:08:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012606732000000001 2024-04-26T01:08:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335000+0000", 2024-04-26T01:08:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00023435799999999999 2024-04-26T01:08:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338932+0000", 2024-04-26T01:08:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0039317500000000003 2024-04-26T01:08:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339070+0000", 2024-04-26T01:08:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000137315 2024-04-26T01:08:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.343357+0000", 2024-04-26T01:08:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042872140000000001 2024-04-26T01:08:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:84 3.5s0 3:a8ed6f44:::smithi16428812-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.390471+0000", 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 25.212933617000001, 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012687102, 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 84 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.622 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.622 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.390471+0000", 2024-04-26T01:08:24.622 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.622 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.622 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.390471+0000", 2024-04-26T01:08:24.622 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.622 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.622 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.390472+0000", 2024-04-26T01:08:24.623 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3200000000000004e-07 2024-04-26T01:08:24.623 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.623 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.623 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.623 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.390475+0000", 2024-04-26T01:08:24.623 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6740000000000001e-06 2024-04-26T01:08:24.623 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.623 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.623 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.623 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.390482+0000", 2024-04-26T01:08:24.623 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6260000000000003e-06 2024-04-26T01:08:24.623 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.624 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.624 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.390488+0000", 2024-04-26T01:08:24.624 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7479999999999999e-06 2024-04-26T01:08:24.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.624 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.624 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394836+0000", 2024-04-26T01:08:24.624 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0043482 2024-04-26T01:08:24.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.625 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:24.625 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394948+0000", 2024-04-26T01:08:24.625 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011179399999999999 2024-04-26T01:08:24.625 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.625 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.625 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.625 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.400983+0000", 2024-04-26T01:08:24.625 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0060351190000000002 2024-04-26T01:08:24.625 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.625 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.625 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.625 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.401097+0000", 2024-04-26T01:08:24.625 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000113773 2024-04-26T01:08:24.625 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.625 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.626 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.626 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.403159+0000", 2024-04-26T01:08:24.626 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0020614359999999998 2024-04-26T01:08:24.626 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.626 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.626 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.626 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.626 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.626 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:128 3.5s0 3:a661dd24:::smithi16428812-43:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:24.626 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:24.626 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 25.146919512, 2024-04-26T01:08:24.626 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.083673129999999998, 2024-04-26T01:08:24.626 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.626 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.627 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:24.627 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.627 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.627 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.627 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 128 2024-04-26T01:08:24.627 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.627 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.627 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.627 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:24.627 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.627 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.627 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.627 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:24.628 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.628 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.628 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.628 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.628 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:24.628 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1299999999999999e-07 2024-04-26T01:08:24.628 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.628 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.628 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.628 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456490+0000", 2024-04-26T01:08:24.628 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4029999999999999e-06 2024-04-26T01:08:24.628 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.628 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.628 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.628 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456501+0000", 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1124999999999999e-05 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456507+0000", 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3180000000000003e-06 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.458775+0000", 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002268061 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.458948+0000", 2024-04-26T01:08:24.629 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017263900000000001 2024-04-26T01:08:24.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.630 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:24.630 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.459498+0000", 2024-04-26T01:08:24.630 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00055037900000000004 2024-04-26T01:08:24.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.630 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:24.630 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460166+0000", 2024-04-26T01:08:24.630 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00066757899999999996 2024-04-26T01:08:24.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.631 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.631 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:24.631 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540012+0000", 2024-04-26T01:08:24.631 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.079846146000000007 2024-04-26T01:08:24.631 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.631 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.631 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.631 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540159+0000", 2024-04-26T01:08:24.631 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014676700000000001 2024-04-26T01:08:24.631 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.631 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.631 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.632 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.632 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.632 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:129 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:24.632 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:24.632 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 25.146890541000001, 2024-04-26T01:08:24.632 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.09570555, 2024-04-26T01:08:24.632 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.632 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.632 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:24.632 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.632 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.632 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.632 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 129 2024-04-26T01:08:24.632 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.633 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.633 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.633 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.633 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:24.633 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.633 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.633 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.633 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.633 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:24.633 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.633 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.633 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.633 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.634 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:24.634 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0699999999999997e-07 2024-04-26T01:08:24.634 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.634 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.634 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.634 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456518+0000", 2024-04-26T01:08:24.634 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.898e-06 2024-04-26T01:08:24.634 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.634 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.634 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.634 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456532+0000", 2024-04-26T01:08:24.635 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3597e-05 2024-04-26T01:08:24.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.635 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.635 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.635 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456537+0000", 2024-04-26T01:08:24.635 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.242e-06 2024-04-26T01:08:24.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.635 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.635 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.635 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.459839+0000", 2024-04-26T01:08:24.635 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033019830000000001 2024-04-26T01:08:24.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.635 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.635 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:24.636 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.459972+0000", 2024-04-26T01:08:24.636 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000133199 2024-04-26T01:08:24.636 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.636 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.636 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.636 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540208+0000", 2024-04-26T01:08:24.636 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.080236319 2024-04-26T01:08:24.636 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.636 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.636 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.636 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540275+0000", 2024-04-26T01:08:24.636 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6486e-05 2024-04-26T01:08:24.637 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.637 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.637 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.637 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.552220+0000", 2024-04-26T01:08:24.637 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011945318999999999 2024-04-26T01:08:24.637 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.637 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.637 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.637 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.637 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.637 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 33/30 ECSubWriteReply(tid=9, last_complete=33'6, committed=1, applied=1))", 2024-04-26T01:08:24.637 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:24.637 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 25.143109466999999, 2024-04-26T01:08:24.637 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.079879818000000005, 2024-04-26T01:08:24.638 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.638 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.638 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:24.638 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.638 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.638 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:24.638 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.638 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.638 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.638 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:24.638 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.639 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.639 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.639 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:24.639 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6599999999999996e-07 2024-04-26T01:08:24.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.639 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.639 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.639 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460301+0000", 2024-04-26T01:08:24.639 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4689999999999999e-06 2024-04-26T01:08:24.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.639 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.639 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.639 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.539830+0000", 2024-04-26T01:08:24.639 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.079528926999999999 2024-04-26T01:08:24.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.640 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.539834+0000", 2024-04-26T01:08:24.640 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5430000000000004e-06 2024-04-26T01:08:24.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.640 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.539887+0000", 2024-04-26T01:08:24.640 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2998999999999999e-05 2024-04-26T01:08:24.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.640 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540175+0000", 2024-04-26T01:08:24.641 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028821400000000002 2024-04-26T01:08:24.641 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.641 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.641 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.641 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.641 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:182 3.5s0 3:a8ed6f44:::smithi16428812-28:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:24.641 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.535158+0000", 2024-04-26T01:08:24.641 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 25.068246813999998, 2024-04-26T01:08:24.641 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015264214, 2024-04-26T01:08:24.641 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.641 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.641 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:24.641 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.641 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.642 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.642 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 182 2024-04-26T01:08:24.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.642 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.642 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.642 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535158+0000", 2024-04-26T01:08:24.642 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.642 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.642 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535158+0000", 2024-04-26T01:08:24.642 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.643 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.643 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535159+0000", 2024-04-26T01:08:24.643 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.71e-07 2024-04-26T01:08:24.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.643 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.643 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535163+0000", 2024-04-26T01:08:24.643 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1540000000000004e-06 2024-04-26T01:08:24.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.643 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.643 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535173+0000", 2024-04-26T01:08:24.643 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.5230000000000001e-06 2024-04-26T01:08:24.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.644 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.644 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535180+0000", 2024-04-26T01:08:24.644 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1629999999999999e-06 2024-04-26T01:08:24.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.644 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.644 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.536667+0000", 2024-04-26T01:08:24.644 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014868850000000001 2024-04-26T01:08:24.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.644 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.644 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.536801+0000", 2024-04-26T01:08:24.644 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013424500000000001 2024-04-26T01:08:24.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.645 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:24.645 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.537375+0000", 2024-04-26T01:08:24.645 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00057366299999999997 2024-04-26T01:08:24.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.645 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:24.645 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.538137+0000", 2024-04-26T01:08:24.645 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00076241899999999997 2024-04-26T01:08:24.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.645 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:24.645 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.550301+0000", 2024-04-26T01:08:24.645 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012163386 2024-04-26T01:08:24.646 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.646 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.646 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.646 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.550422+0000", 2024-04-26T01:08:24.646 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012190500000000001 2024-04-26T01:08:24.646 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.646 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.646 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.646 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.647 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.647 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:183 3.5s0 3:a8ed6f44:::smithi16428812-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:24.647 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:24.647 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 25.068211478999999, 2024-04-26T01:08:24.647 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.017578818999999999, 2024-04-26T01:08:24.647 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.647 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.647 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:24.647 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.647 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.647 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.647 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 183 2024-04-26T01:08:24.648 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.648 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.648 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.648 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.648 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:24.648 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.648 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.648 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.648 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.648 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:24.648 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.648 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.649 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.649 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.649 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:24.649 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9500000000000002e-07 2024-04-26T01:08:24.649 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.649 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.649 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.649 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535198+0000", 2024-04-26T01:08:24.649 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6430000000000001e-06 2024-04-26T01:08:24.649 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.649 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.649 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.649 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535207+0000", 2024-04-26T01:08:24.650 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9730000000000003e-06 2024-04-26T01:08:24.650 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.650 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.650 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.650 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535215+0000", 2024-04-26T01:08:24.650 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1540000000000002e-06 2024-04-26T01:08:24.650 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.650 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.650 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.650 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.537794+0000", 2024-04-26T01:08:24.650 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0025788740000000001 2024-04-26T01:08:24.651 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.651 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:24.651 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.537868+0000", 2024-04-26T01:08:24.651 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4028000000000003e-05 2024-04-26T01:08:24.651 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.651 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.651 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.550967+0000", 2024-04-26T01:08:24.651 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013099006999999999 2024-04-26T01:08:24.651 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.651 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.651 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551052+0000", 2024-04-26T01:08:24.652 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4667999999999996e-05 2024-04-26T01:08:24.652 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.652 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.652 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.652 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.552772+0000", 2024-04-26T01:08:24.652 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0017207769999999999 2024-04-26T01:08:24.652 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.652 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.652 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.652 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.652 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.652 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 33/30 ECSubWriteReply(tid=14, last_complete=33'9, committed=1, applied=1))", 2024-04-26T01:08:24.652 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.551733+0000", 2024-04-26T01:08:24.653 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 25.051672018000001, 2024-04-26T01:08:24.653 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.40949085299999999, 2024-04-26T01:08:24.653 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.653 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.653 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:24.653 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.653 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.653 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.653 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551733+0000", 2024-04-26T01:08:24.653 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.653 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.653 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.653 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.653 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551733+0000", 2024-04-26T01:08:24.654 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.654 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.654 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.654 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.654 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551734+0000", 2024-04-26T01:08:24.654 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7199999999999998e-07 2024-04-26T01:08:24.654 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.654 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.654 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.654 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551739+0000", 2024-04-26T01:08:24.654 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9520000000000002e-06 2024-04-26T01:08:24.654 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.654 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.655 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.655 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.960976+0000", 2024-04-26T01:08:24.655 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.40923691400000001 2024-04-26T01:08:24.655 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.655 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.655 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.655 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.960989+0000", 2024-04-26T01:08:24.655 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3536000000000001e-05 2024-04-26T01:08:24.655 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.655 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.655 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.655 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.961157+0000", 2024-04-26T01:08:24.655 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016794600000000001 2024-04-26T01:08:24.656 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.656 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.656 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.656 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.961224+0000", 2024-04-26T01:08:24.656 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6733e-05 2024-04-26T01:08:24.656 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.656 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.656 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.656 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.656 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.656 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:251 3.5s0 3:acf4d671:::smithi16428812-10:head [append 0~0] snapc 4=[4,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:24.656 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:03.428674+0000", 2024-04-26T01:08:24.656 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 21.174730580999999, 2024-04-26T01:08:24.656 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.48188827299999998, 2024-04-26T01:08:24.657 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.657 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.657 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:24.657 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.657 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.657 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.657 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 251 2024-04-26T01:08:24.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.657 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.657 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.657 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428674+0000", 2024-04-26T01:08:24.657 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.658 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428674+0000", 2024-04-26T01:08:24.658 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.658 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428676+0000", 2024-04-26T01:08:24.658 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6559999999999999e-06 2024-04-26T01:08:24.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.658 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428685+0000", 2024-04-26T01:08:24.658 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0869999999999996e-06 2024-04-26T01:08:24.659 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.659 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.659 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.659 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428707+0000", 2024-04-26T01:08:24.659 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1365999999999998e-05 2024-04-26T01:08:24.659 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.659 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.659 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.659 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428723+0000", 2024-04-26T01:08:24.659 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6563e-05 2024-04-26T01:08:24.659 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.659 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.659 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.659 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.434334+0000", 2024-04-26T01:08:24.660 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056104170000000004 2024-04-26T01:08:24.660 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.660 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.660 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.434677+0000", 2024-04-26T01:08:24.660 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000343937 2024-04-26T01:08:24.660 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.660 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:24.660 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.436169+0000", 2024-04-26T01:08:24.660 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014915939999999999 2024-04-26T01:08:24.660 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.661 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:24.661 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.438266+0000", 2024-04-26T01:08:24.661 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002096746 2024-04-26T01:08:24.661 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.661 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.661 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:24.661 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.910445+0000", 2024-04-26T01:08:24.661 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.47217926300000002 2024-04-26T01:08:24.661 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.661 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.661 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.661 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.910563+0000", 2024-04-26T01:08:24.661 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000117644 2024-04-26T01:08:24.661 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.661 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.662 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.662 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.662 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.662 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:412 3.5s0 3:a661dd24:::smithi16428812-43:head [append 0~688128] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-26T01:08:24.662 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.437736+0000", 2024-04-26T01:08:24.662 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 14.165669133, 2024-04-26T01:08:24.662 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015629959999999998, 2024-04-26T01:08:24.662 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.662 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.662 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:24.662 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.662 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.663 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.663 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 412 2024-04-26T01:08:24.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.663 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.663 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.663 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.663 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.437736+0000", 2024-04-26T01:08:24.663 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.663 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.663 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.663 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.437736+0000", 2024-04-26T01:08:24.663 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.664 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.664 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.437739+0000", 2024-04-26T01:08:24.664 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1530000000000001e-06 2024-04-26T01:08:24.664 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.664 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.664 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.438458+0000", 2024-04-26T01:08:24.664 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00071916899999999997 2024-04-26T01:08:24.664 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.664 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.664 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.438496+0000", 2024-04-26T01:08:24.665 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8022999999999998e-05 2024-04-26T01:08:24.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.665 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.665 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.665 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.438516+0000", 2024-04-26T01:08:24.665 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9936e-05 2024-04-26T01:08:24.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.665 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.665 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.665 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.438651+0000", 2024-04-26T01:08:24.665 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013456899999999999 2024-04-26T01:08:24.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.665 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.665 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2024-04-26T01:08:24.666 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.438661+0000", 2024-04-26T01:08:24.666 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0363000000000001e-05 2024-04-26T01:08:24.666 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.666 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.666 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.666 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.442159+0000", 2024-04-26T01:08:24.666 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0034977509999999999 2024-04-26T01:08:24.666 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.666 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.666 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.666 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.442304+0000", 2024-04-26T01:08:24.666 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014504599999999999 2024-04-26T01:08:24.666 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.666 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.667 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:24.667 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.444152+0000", 2024-04-26T01:08:24.667 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0018484669999999999 2024-04-26T01:08:24.667 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.667 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.667 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:24.667 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.452613+0000", 2024-04-26T01:08:24.667 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0084610940000000006 2024-04-26T01:08:24.667 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.667 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.667 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:24.667 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.453249+0000", 2024-04-26T01:08:24.667 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000635989 2024-04-26T01:08:24.668 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.668 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.668 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.668 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.453366+0000", 2024-04-26T01:08:24.668 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0001164 2024-04-26T01:08:24.668 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.668 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.668 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.668 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.668 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.668 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:413 3.5s0 3:a661dd24:::smithi16428812-43:head [append 0~540672] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-26T01:08:24.668 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.442205+0000", 2024-04-26T01:08:24.669 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 14.161199693, 2024-04-26T01:08:24.669 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011472952, 2024-04-26T01:08:24.669 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.669 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.669 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:24.669 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.669 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.669 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.669 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 413 2024-04-26T01:08:24.669 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.669 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.669 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.670 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.670 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.442205+0000", 2024-04-26T01:08:24.670 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.670 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.670 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.670 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.670 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.442205+0000", 2024-04-26T01:08:24.670 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.670 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.670 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.670 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.670 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.442209+0000", 2024-04-26T01:08:24.670 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7730000000000001e-06 2024-04-26T01:08:24.671 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.671 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.671 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.442992+0000", 2024-04-26T01:08:24.671 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00078293099999999995 2024-04-26T01:08:24.671 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.671 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.671 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443016+0000", 2024-04-26T01:08:24.671 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.376e-05 2024-04-26T01:08:24.671 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.671 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.672 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443033+0000", 2024-04-26T01:08:24.672 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7479e-05 2024-04-26T01:08:24.672 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.672 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.672 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.672 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.446209+0000", 2024-04-26T01:08:24.672 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0031761599999999999 2024-04-26T01:08:24.672 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.672 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.672 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.672 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.446534+0000", 2024-04-26T01:08:24.672 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00032445500000000002 2024-04-26T01:08:24.673 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.673 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.673 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:24.673 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.448373+0000", 2024-04-26T01:08:24.673 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0018392580000000001 2024-04-26T01:08:24.673 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.673 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.673 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:24.673 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.452615+0000", 2024-04-26T01:08:24.673 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042421230000000004 2024-04-26T01:08:24.674 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.674 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.674 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:24.674 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.453618+0000", 2024-04-26T01:08:24.674 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010031319999999999 2024-04-26T01:08:24.674 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.674 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.674 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.674 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.453678+0000", 2024-04-26T01:08:24.674 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9880999999999998e-05 2024-04-26T01:08:24.674 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.674 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.674 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.675 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.675 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.675 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:414 3.5s0 3:a661dd24:::smithi16428812-43:head [append 0~229376] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-26T01:08:24.675 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.443547+0000", 2024-04-26T01:08:24.675 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 14.159857874, 2024-04-26T01:08:24.675 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011928169000000001, 2024-04-26T01:08:24.675 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.675 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.675 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:24.675 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.675 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.675 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.675 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 414 2024-04-26T01:08:24.675 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.676 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.676 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.676 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.676 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443547+0000", 2024-04-26T01:08:24.676 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.676 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.676 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.676 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.676 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443547+0000", 2024-04-26T01:08:24.676 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.676 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.676 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.676 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.676 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443550+0000", 2024-04-26T01:08:24.677 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0060000000000001e-06 2024-04-26T01:08:24.677 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.677 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.677 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.677 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443876+0000", 2024-04-26T01:08:24.677 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00032578199999999999 2024-04-26T01:08:24.677 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.677 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.677 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.677 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443897+0000", 2024-04-26T01:08:24.677 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1433e-05 2024-04-26T01:08:24.677 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.677 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.677 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.678 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443910+0000", 2024-04-26T01:08:24.678 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2969000000000001e-05 2024-04-26T01:08:24.678 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.678 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.678 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.678 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.449751+0000", 2024-04-26T01:08:24.678 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0058404659999999999 2024-04-26T01:08:24.678 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.678 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.678 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.678 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.449999+0000", 2024-04-26T01:08:24.678 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024795900000000002 2024-04-26T01:08:24.678 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.679 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.679 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:24.679 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.451445+0000", 2024-04-26T01:08:24.679 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001446391 2024-04-26T01:08:24.679 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.679 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.679 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:24.679 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.453555+0000", 2024-04-26T01:08:24.679 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021102809999999999 2024-04-26T01:08:24.679 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.679 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.679 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:24.679 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.455407+0000", 2024-04-26T01:08:24.679 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0018518059999999999 2024-04-26T01:08:24.680 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.680 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.680 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.680 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.455475+0000", 2024-04-26T01:08:24.680 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8076000000000003e-05 2024-04-26T01:08:24.680 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.680 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.680 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.680 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.680 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.680 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:415 3.5s0 3:a661dd24:::smithi16428812-43:head [setxattr _header (56) in=7b] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-26T01:08:24.680 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.443941+0000", 2024-04-26T01:08:24.680 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 14.159464534, 2024-04-26T01:08:24.680 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.016144703999999999, 2024-04-26T01:08:24.681 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.681 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.681 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:24.681 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.681 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.681 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.681 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 415 2024-04-26T01:08:24.681 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.681 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.681 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.681 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.681 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443941+0000", 2024-04-26T01:08:24.681 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.682 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.682 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.682 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.682 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443941+0000", 2024-04-26T01:08:24.682 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.682 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.682 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.682 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.682 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443942+0000", 2024-04-26T01:08:24.682 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1999999999999999e-06 2024-04-26T01:08:24.682 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.682 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.683 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.683 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443950+0000", 2024-04-26T01:08:24.683 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.0639999999999994e-06 2024-04-26T01:08:24.683 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.683 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.683 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.683 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443961+0000", 2024-04-26T01:08:24.683 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0798999999999999e-05 2024-04-26T01:08:24.683 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.683 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.683 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.683 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443968+0000", 2024-04-26T01:08:24.683 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6699999999999994e-06 2024-04-26T01:08:24.683 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.683 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.684 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.684 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.453783+0000", 2024-04-26T01:08:24.684 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0098143120000000004 2024-04-26T01:08:24.684 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.684 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.684 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.684 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.453843+0000", 2024-04-26T01:08:24.684 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0201000000000002e-05 2024-04-26T01:08:24.684 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.684 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.684 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:24.684 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.454111+0000", 2024-04-26T01:08:24.684 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00026851700000000002 2024-04-26T01:08:24.684 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.684 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.685 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:24.685 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.455183+0000", 2024-04-26T01:08:24.685 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010722030000000001 2024-04-26T01:08:24.685 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.685 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.685 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:24.685 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.459986+0000", 2024-04-26T01:08:24.685 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0048022350000000002 2024-04-26T01:08:24.685 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.685 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.685 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.685 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.460085+0000", 2024-04-26T01:08:24.685 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9502999999999999e-05 2024-04-26T01:08:24.685 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.685 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.686 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.686 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.686 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.686 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:416 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-26T01:08:24.686 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.443980+0000", 2024-04-26T01:08:24.686 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 14.159425191, 2024-04-26T01:08:24.686 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.018880322000000001, 2024-04-26T01:08:24.686 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.686 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.686 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:24.686 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.686 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.686 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.686 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 416 2024-04-26T01:08:24.687 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.687 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.687 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.687 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443980+0000", 2024-04-26T01:08:24.687 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.687 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.687 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.687 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443980+0000", 2024-04-26T01:08:24.687 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.687 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.687 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.687 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443981+0000", 2024-04-26T01:08:24.688 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7000000000000003e-07 2024-04-26T01:08:24.688 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.688 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.688 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.688 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443984+0000", 2024-04-26T01:08:24.688 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.36e-06 2024-04-26T01:08:24.688 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.688 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.688 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.688 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443994+0000", 2024-04-26T01:08:24.688 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0285999999999999e-05 2024-04-26T01:08:24.688 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.688 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.688 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.688 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.444002+0000", 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4970000000000004e-06 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.454900+0000", 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010897784000000001 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.455102+0000", 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00020183000000000001 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.460460+0000", 2024-04-26T01:08:24.689 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0053580140000000004 2024-04-26T01:08:24.690 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.690 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.690 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.690 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.460760+0000", 2024-04-26T01:08:24.690 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00030007900000000002 2024-04-26T01:08:24.690 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.690 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.690 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.690 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.462860+0000", 2024-04-26T01:08:24.690 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021005020000000002 2024-04-26T01:08:24.690 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.690 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.690 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.691 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.691 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.691 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.3s3 44/30 ECSubWrite(tid=198, reqid=client.4521.0:420, at_version=44'13, trim_to=0'0, roll_forward_to=44'11))", 2024-04-26T01:08:24.691 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.466337+0000", 2024-04-26T01:08:24.691 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 14.137068434, 2024-04-26T01:08:24.691 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.57864738100000002, 2024-04-26T01:08:24.691 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.691 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.691 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:24.691 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.691 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.691 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.691 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466337+0000", 2024-04-26T01:08:24.691 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.692 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.692 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.692 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.692 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466337+0000", 2024-04-26T01:08:24.692 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.692 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.692 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.692 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.692 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466338+0000", 2024-04-26T01:08:24.692 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4479999999999999e-06 2024-04-26T01:08:24.692 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.692 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.692 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.692 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466374+0000", 2024-04-26T01:08:24.692 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6137999999999999e-05 2024-04-26T01:08:24.693 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.693 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.693 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.693 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.043926+0000", 2024-04-26T01:08:24.693 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.57755188800000001 2024-04-26T01:08:24.693 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.693 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.693 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.693 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.043944+0000", 2024-04-26T01:08:24.693 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7745999999999999e-05 2024-04-26T01:08:24.693 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.693 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.693 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.693 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044049+0000", 2024-04-26T01:08:24.694 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010500700000000001 2024-04-26T01:08:24.694 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.694 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.694 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:24.694 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044068+0000", 2024-04-26T01:08:24.694 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8984999999999999e-05 2024-04-26T01:08:24.694 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.694 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.694 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:24.694 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044949+0000", 2024-04-26T01:08:24.694 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00088138400000000003 2024-04-26T01:08:24.694 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.694 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.694 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.695 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044984+0000", 2024-04-26T01:08:24.695 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4785e-05 2024-04-26T01:08:24.695 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.695 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.695 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.695 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.695 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.695 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s4 45/44 ECSubWrite(tid=204, reqid=client.4521.0:364, at_version=45'18, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:24.695 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:11.085534+0000", 2024-04-26T01:08:24.695 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 13.517870636, 2024-04-26T01:08:24.695 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011269358, 2024-04-26T01:08:24.695 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.695 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.695 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:24.695 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.696 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.696 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.696 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.085534+0000", 2024-04-26T01:08:24.696 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.696 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.696 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.696 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.696 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.085534+0000", 2024-04-26T01:08:24.696 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.696 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.696 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.696 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.696 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.085536+0000", 2024-04-26T01:08:24.696 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.15e-06 2024-04-26T01:08:24.697 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.697 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.697 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.697 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.085702+0000", 2024-04-26T01:08:24.697 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016618399999999999 2024-04-26T01:08:24.697 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.697 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.697 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.697 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.094903+0000", 2024-04-26T01:08:24.697 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0092013500000000005 2024-04-26T01:08:24.697 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.697 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.697 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.697 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.094926+0000", 2024-04-26T01:08:24.697 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3133e-05 2024-04-26T01:08:24.698 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.698 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.698 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.698 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.095048+0000", 2024-04-26T01:08:24.698 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000121722 2024-04-26T01:08:24.698 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.698 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.698 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:24.698 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.095057+0000", 2024-04-26T01:08:24.698 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3319999999999998e-06 2024-04-26T01:08:24.698 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.698 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.698 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:24.698 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.096713+0000", 2024-04-26T01:08:24.699 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0016560310000000001 2024-04-26T01:08:24.699 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.699 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.699 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.699 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.096804+0000", 2024-04-26T01:08:24.699 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0456000000000003e-05 2024-04-26T01:08:24.699 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.699 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.699 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.699 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.699 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.699 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:554 3.5s0 3:a661dd24:::smithi16428812-43:head [copy-from ver 26] snapc e=[e,d,b,7] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-04-26T01:08:24.699 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:21.094110+0000", 2024-04-26T01:08:24.699 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 3.5092946449999998, 2024-04-26T01:08:24.700 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.050017671, 2024-04-26T01:08:24.700 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.700 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.700 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:24.700 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.700 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.700 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.700 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 554 2024-04-26T01:08:24.700 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.700 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.700 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.700 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.700 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094110+0000", 2024-04-26T01:08:24.700 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.701 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.701 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.701 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.701 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094110+0000", 2024-04-26T01:08:24.701 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.701 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.701 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.701 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.701 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094114+0000", 2024-04-26T01:08:24.701 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.89e-06 2024-04-26T01:08:24.701 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.701 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.701 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.701 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094134+0000", 2024-04-26T01:08:24.702 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0007e-05 2024-04-26T01:08:24.702 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.702 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.702 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.702 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094181+0000", 2024-04-26T01:08:24.702 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6731999999999999e-05 2024-04-26T01:08:24.702 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.702 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.702 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.702 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094205+0000", 2024-04-26T01:08:24.702 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4326999999999999e-05 2024-04-26T01:08:24.702 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.702 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.702 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.702 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094455+0000", 2024-04-26T01:08:24.703 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024929999999999999 2024-04-26T01:08:24.703 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.703 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.703 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2024-04-26T01:08:24.703 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094477+0000", 2024-04-26T01:08:24.703 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2646e-05 2024-04-26T01:08:24.703 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.703 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.703 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.703 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105141+0000", 2024-04-26T01:08:24.703 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010664167 2024-04-26T01:08:24.703 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.703 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.703 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.703 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105392+0000", 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00025070700000000002 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.135272+0000", 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.029880263000000001 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.142569+0000", 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0072964270000000003 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.142755+0000", 2024-04-26T01:08:24.704 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00018603399999999999 2024-04-26T01:08:24.705 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.705 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.705 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.705 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.144128+0000", 2024-04-26T01:08:24.705 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0013731710000000001 2024-04-26T01:08:24.705 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.705 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.705 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.705 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.705 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.705 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:555 3.5s0 3:a661dd24:::smithi16428812-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e55)", 2024-04-26T01:08:24.705 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:21.094258+0000", 2024-04-26T01:08:24.705 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 3.5091474800000002, 2024-04-26T01:08:24.705 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.045831917999999999, 2024-04-26T01:08:24.706 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:24.706 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:24.706 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:24.706 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:24.706 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:24.706 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:24.706 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 555 2024-04-26T01:08:24.706 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.706 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:24.706 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.706 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:24.706 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094258+0000", 2024-04-26T01:08:24.706 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.707 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:24.707 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094258+0000", 2024-04-26T01:08:24.707 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:24.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.707 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:24.707 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094259+0000", 2024-04-26T01:08:24.707 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7370000000000001e-06 2024-04-26T01:08:24.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.707 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:24.707 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094268+0000", 2024-04-26T01:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8359999999999998e-06 2024-04-26T01:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094289+0000", 2024-04-26T01:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0937e-05 2024-04-26T01:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094299+0000", 2024-04-26T01:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.029e-05 2024-04-26T01:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094609+0000", 2024-04-26T01:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00031005900000000001 2024-04-26T01:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094619+0000", 2024-04-26T01:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9799999999999993e-06 2024-04-26T01:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105868+0000", 2024-04-26T01:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011248555 2024-04-26T01:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2024-04-26T01:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105946+0000", 2024-04-26T01:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8005999999999998e-05 2024-04-26T01:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.139672+0000", 2024-04-26T01:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.033725805999999997 2024-04-26T01:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.139869+0000", 2024-04-26T01:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019683 2024-04-26T01:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.140089+0000", 2024-04-26T01:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022088199999999999 2024-04-26T01:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:24.713 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:08:24.713 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:24.709+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:24.813 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:24.808+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:24.848 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:24.848 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:24.848 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-04-26T01:08:24.849 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:24.863 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:08:24.950 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:24.948+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:25.015 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:25.016+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:25.034 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:25.034 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:25.034 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-04-26T01:08:25.034 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-04-26T01:08:25.035 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:25.048 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:08:25.115 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:25.116+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:25.157 INFO:tasks.rados.rados.0.smithi164.stdout:238: done (4 left) 2024-04-26T01:08:25.157 INFO:tasks.rados.rados.0.smithi164.stdout:239: setattr oid 30 current snap is 14 2024-04-26T01:08:25.158 INFO:tasks.rados.rados.0.smithi164.stdout:235: finishing copy_from to smithi16428812-38 2024-04-26T01:08:25.158 INFO:tasks.rados.rados.0.smithi164.stdout:235: got expected ENOENT (src dne) 2024-04-26T01:08:25.158 INFO:tasks.rados.rados.0.smithi164.stdout:235: finishing copy_from racing read to smithi16428812-38 2024-04-26T01:08:25.158 INFO:tasks.rados.rados.0.smithi164.stdout:236: expect (ObjNum 98 snap 11 seq_num 98) 2024-04-26T01:08:25.194 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:25.196+0000 7ff27e000640 -1 osd.11 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:08:25.216 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:25.216+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:25.231 INFO:tasks.rados.rados.0.smithi164.stdout:234: finishing copy_from racing read to smithi16428812-40 2024-04-26T01:08:25.232 INFO:tasks.rados.rados.0.smithi164.stdout:237: expect (ObjNum 108 snap 13 seq_num 108) 2024-04-26T01:08:25.235 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:25.235 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:25.235 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:25.235 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:25.236 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.236 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.7s3 33/30 ECSubWrite(tid=9, reqid=client.4521.0:38, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:25.236 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.325556+0000", 2024-04-26T01:08:25.236 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 25.908053196000001, 2024-04-26T01:08:25.236 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.028941706000000001, 2024-04-26T01:08:25.236 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.236 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.236 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:25.236 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.236 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.236 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.236 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325556+0000", 2024-04-26T01:08:25.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.237 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.237 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.237 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325556+0000", 2024-04-26T01:08:25.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.237 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.237 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.237 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325557+0000", 2024-04-26T01:08:25.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.221e-06 2024-04-26T01:08:25.237 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.237 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.238 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325581+0000", 2024-04-26T01:08:25.238 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3502e-05 2024-04-26T01:08:25.238 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.238 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.238 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.238 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353123+0000", 2024-04-26T01:08:25.238 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.027542540000000001 2024-04-26T01:08:25.238 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.238 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.238 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.238 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353131+0000", 2024-04-26T01:08:25.238 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6909999999999997e-06 2024-04-26T01:08:25.238 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.239 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.239 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.239 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353823+0000", 2024-04-26T01:08:25.239 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00069229100000000004 2024-04-26T01:08:25.239 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.239 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.239 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.239 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353835+0000", 2024-04-26T01:08:25.239 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1507e-05 2024-04-26T01:08:25.239 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.239 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.239 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.239 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.354460+0000", 2024-04-26T01:08:25.240 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00062488599999999995 2024-04-26T01:08:25.240 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.240 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.240 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.240 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.354498+0000", 2024-04-26T01:08:25.240 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8068000000000002e-05 2024-04-26T01:08:25.240 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.240 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.240 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.240 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.240 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.240 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:43 3.6s0 3:61b5d9f4:::smithi16428812-15:head [append 0~245760,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:25.240 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.344935+0000", 2024-04-26T01:08:25.241 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 25.888674407, 2024-04-26T01:08:25.241 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012248536000000001, 2024-04-26T01:08:25.241 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.241 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.241 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:25.241 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.241 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.241 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.241 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 43 2024-04-26T01:08:25.241 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.241 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.241 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.241 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.241 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.344935+0000", 2024-04-26T01:08:25.242 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.242 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.242 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.242 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.344935+0000", 2024-04-26T01:08:25.242 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.242 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.242 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.242 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.344937+0000", 2024-04-26T01:08:25.242 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9750000000000001e-06 2024-04-26T01:08:25.242 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.243 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.243 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.243 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345580+0000", 2024-04-26T01:08:25.243 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00064293 2024-04-26T01:08:25.243 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.243 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.243 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.243 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345596+0000", 2024-04-26T01:08:25.243 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6334e-05 2024-04-26T01:08:25.243 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.243 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.243 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.244 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345605+0000", 2024-04-26T01:08:25.244 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2080000000000006e-06 2024-04-26T01:08:25.244 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.244 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.244 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.244 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345700+0000", 2024-04-26T01:08:25.244 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4698999999999996e-05 2024-04-26T01:08:25.244 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.244 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.244 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.244 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345954+0000", 2024-04-26T01:08:25.244 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00025357 2024-04-26T01:08:25.244 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.245 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.245 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.245 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.347095+0000", 2024-04-26T01:08:25.245 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001140978 2024-04-26T01:08:25.245 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.245 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.245 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.245 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349634+0000", 2024-04-26T01:08:25.245 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0025389969999999999 2024-04-26T01:08:25.245 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.245 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.245 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:25.246 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.357101+0000", 2024-04-26T01:08:25.246 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0074677579999999997 2024-04-26T01:08:25.246 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.246 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.246 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.246 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.357183+0000", 2024-04-26T01:08:25.246 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2087000000000002e-05 2024-04-26T01:08:25.246 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.246 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.246 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.246 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.246 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.246 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:44 3.6s0 3:61b5d9f4:::smithi16428812-15:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:25.247 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.345630+0000", 2024-04-26T01:08:25.247 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 25.887979055999999, 2024-04-26T01:08:25.247 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011979597999999999, 2024-04-26T01:08:25.247 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.247 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.247 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:25.247 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.247 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.247 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.247 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 44 2024-04-26T01:08:25.247 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.247 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.247 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.247 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.247 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345630+0000", 2024-04-26T01:08:25.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.248 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.248 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.248 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345630+0000", 2024-04-26T01:08:25.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.248 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.248 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.248 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345631+0000", 2024-04-26T01:08:25.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7999999999999995e-07 2024-04-26T01:08:25.248 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.249 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.249 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345634+0000", 2024-04-26T01:08:25.249 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.32e-06 2024-04-26T01:08:25.249 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.249 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.249 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.249 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345643+0000", 2024-04-26T01:08:25.249 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2130000000000008e-06 2024-04-26T01:08:25.249 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.249 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.249 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.249 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345649+0000", 2024-04-26T01:08:25.249 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.127e-06 2024-04-26T01:08:25.249 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.250 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.250 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.348399+0000", 2024-04-26T01:08:25.250 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027505519999999999 2024-04-26T01:08:25.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.250 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.250 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.348498+0000", 2024-04-26T01:08:25.250 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9272999999999996e-05 2024-04-26T01:08:25.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.250 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.251 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.348992+0000", 2024-04-26T01:08:25.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000493444 2024-04-26T01:08:25.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.251 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.251 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349687+0000", 2024-04-26T01:08:25.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00069528500000000002 2024-04-26T01:08:25.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.251 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:25.251 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.357516+0000", 2024-04-26T01:08:25.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0078283520000000002 2024-04-26T01:08:25.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.252 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.252 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.357610+0000", 2024-04-26T01:08:25.252 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4351999999999994e-05 2024-04-26T01:08:25.252 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.252 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.252 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.252 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.252 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.252 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:45 3.6s0 3:61b5d9f4:::smithi16428812-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:25.252 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.345679+0000", 2024-04-26T01:08:25.252 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 25.887930346000001, 2024-04-26T01:08:25.252 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014982359000000001, 2024-04-26T01:08:25.252 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.252 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.253 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.253 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.253 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.253 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.253 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 45 2024-04-26T01:08:25.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.253 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.253 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.253 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345679+0000", 2024-04-26T01:08:25.253 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345679+0000", 2024-04-26T01:08:25.254 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345679+0000", 2024-04-26T01:08:25.254 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9599999999999999e-07 2024-04-26T01:08:25.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345683+0000", 2024-04-26T01:08:25.255 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0110000000000001e-06 2024-04-26T01:08:25.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.255 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.255 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.255 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345696+0000", 2024-04-26T01:08:25.255 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3557000000000001e-05 2024-04-26T01:08:25.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.255 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.255 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.255 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345701+0000", 2024-04-26T01:08:25.255 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6190000000000002e-06 2024-04-26T01:08:25.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.255 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.256 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.256 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349537+0000", 2024-04-26T01:08:25.256 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0038363500000000001 2024-04-26T01:08:25.256 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.256 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.256 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:25.256 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349626+0000", 2024-04-26T01:08:25.256 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8757000000000004e-05 2024-04-26T01:08:25.256 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.256 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.256 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.256 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.358038+0000", 2024-04-26T01:08:25.256 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0084123970000000003 2024-04-26T01:08:25.256 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.257 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.257 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.257 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.358133+0000", 2024-04-26T01:08:25.257 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4949000000000002e-05 2024-04-26T01:08:25.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.257 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.257 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.257 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.360661+0000", 2024-04-26T01:08:25.257 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002528223 2024-04-26T01:08:25.257 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.257 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.257 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.257 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.258 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.fs6 33/30 ECSubWrite(tid=13, reqid=client.4521.0:53, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:25.258 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.352280+0000", 2024-04-26T01:08:25.258 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 25.881329757, 2024-04-26T01:08:25.258 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.017609129000000001, 2024-04-26T01:08:25.258 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.258 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.258 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:25.258 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.258 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.258 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.258 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.352280+0000", 2024-04-26T01:08:25.258 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.258 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.258 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.259 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.259 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.352280+0000", 2024-04-26T01:08:25.259 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.259 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.259 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.259 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.259 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.352280+0000", 2024-04-26T01:08:25.259 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6400000000000001e-07 2024-04-26T01:08:25.259 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.259 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.259 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.259 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.352302+0000", 2024-04-26T01:08:25.259 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1577e-05 2024-04-26T01:08:25.259 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.259 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.260 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.260 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369217+0000", 2024-04-26T01:08:25.260 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.016914825000000001 2024-04-26T01:08:25.260 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.260 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.260 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.260 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369224+0000", 2024-04-26T01:08:25.260 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9720000000000003e-06 2024-04-26T01:08:25.260 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.260 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.260 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.260 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369277+0000", 2024-04-26T01:08:25.260 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2924999999999999e-05 2024-04-26T01:08:25.260 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.261 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.261 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.261 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369284+0000", 2024-04-26T01:08:25.261 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0840000000000003e-06 2024-04-26T01:08:25.261 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.261 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.261 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.261 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369847+0000", 2024-04-26T01:08:25.261 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00056294600000000004 2024-04-26T01:08:25.261 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.261 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.261 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.261 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369889+0000", 2024-04-26T01:08:25.262 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1936e-05 2024-04-26T01:08:25.262 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.262 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.262 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.262 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.262 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.262 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.7s3 33/30 ECSubWrite(tid=16, reqid=client.4521.0:65, at_version=33'8, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:08:25.262 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.366487+0000", 2024-04-26T01:08:25.262 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 25.867121869999998, 2024-04-26T01:08:25.262 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.079262088999999994, 2024-04-26T01:08:25.262 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.262 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.262 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:25.263 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.263 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.263 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.263 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366487+0000", 2024-04-26T01:08:25.263 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.263 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.263 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.263 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.263 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366487+0000", 2024-04-26T01:08:25.263 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.263 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.263 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.263 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.263 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366488+0000", 2024-04-26T01:08:25.264 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6700000000000002e-07 2024-04-26T01:08:25.264 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.264 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.264 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.264 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366510+0000", 2024-04-26T01:08:25.264 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1591999999999999e-05 2024-04-26T01:08:25.264 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.264 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.264 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.264 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444763+0000", 2024-04-26T01:08:25.264 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.078252639999999998 2024-04-26T01:08:25.264 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.264 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.265 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.265 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444773+0000", 2024-04-26T01:08:25.265 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0134e-05 2024-04-26T01:08:25.265 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.265 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.265 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.265 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444870+0000", 2024-04-26T01:08:25.265 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6847000000000003e-05 2024-04-26T01:08:25.265 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.265 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.265 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.265 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444880+0000", 2024-04-26T01:08:25.265 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0312999999999999e-05 2024-04-26T01:08:25.265 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.266 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.266 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.266 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.445715+0000", 2024-04-26T01:08:25.266 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00083498700000000003 2024-04-26T01:08:25.266 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.266 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.266 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.266 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.445750+0000", 2024-04-26T01:08:25.266 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4609e-05 2024-04-26T01:08:25.266 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.266 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.266 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.266 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.266 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.267 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:72 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:25.267 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.372400+0000", 2024-04-26T01:08:25.267 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 25.861209340999999, 2024-04-26T01:08:25.267 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018232822999999999, 2024-04-26T01:08:25.267 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.267 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.267 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.267 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.267 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.267 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.267 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 72 2024-04-26T01:08:25.267 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.267 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.268 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.268 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.268 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372400+0000", 2024-04-26T01:08:25.268 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.268 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.268 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.268 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.268 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372400+0000", 2024-04-26T01:08:25.268 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.268 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.268 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.268 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.268 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372401+0000", 2024-04-26T01:08:25.269 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7899999999999998e-07 2024-04-26T01:08:25.269 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.269 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.269 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.269 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372404+0000", 2024-04-26T01:08:25.269 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0469999999999998e-06 2024-04-26T01:08:25.269 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.269 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.269 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.269 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372413+0000", 2024-04-26T01:08:25.269 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3729999999999999e-06 2024-04-26T01:08:25.269 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.269 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.269 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.270 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372419+0000", 2024-04-26T01:08:25.270 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1070000000000001e-06 2024-04-26T01:08:25.270 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.270 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.270 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.270 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374838+0000", 2024-04-26T01:08:25.270 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024188289999999999 2024-04-26T01:08:25.270 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.270 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.270 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:25.270 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374919+0000", 2024-04-26T01:08:25.270 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0717000000000007e-05 2024-04-26T01:08:25.270 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.270 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.271 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.271 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.381324+0000", 2024-04-26T01:08:25.271 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.006405666 2024-04-26T01:08:25.271 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.271 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.271 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.271 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.381402+0000", 2024-04-26T01:08:25.271 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7323000000000005e-05 2024-04-26T01:08:25.271 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.271 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.271 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.271 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.390633+0000", 2024-04-26T01:08:25.271 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0092311819999999992 2024-04-26T01:08:25.271 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.272 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.272 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.272 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.272 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.272 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpRead(3.cs2 33/30 ECSubRead(tid=19, to_read={3:3ca9ce7a:::smithi16428812-23:head=0,4096,0}, subchunks={3:3ca9ce7a:::smithi16428812-23:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:25.272 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.374332+0000", 2024-04-26T01:08:25.272 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 25.859277248000001, 2024-04-26T01:08:25.272 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.023990841999999998, 2024-04-26T01:08:25.272 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.272 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.272 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:25.272 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.272 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.273 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.273 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374332+0000", 2024-04-26T01:08:25.273 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.273 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.273 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.273 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.273 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374332+0000", 2024-04-26T01:08:25.273 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.273 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.273 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.273 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.273 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374333+0000", 2024-04-26T01:08:25.273 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0070000000000001e-06 2024-04-26T01:08:25.273 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.273 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.274 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.274 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374345+0000", 2024-04-26T01:08:25.274 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1853e-05 2024-04-26T01:08:25.274 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.274 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.274 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.274 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398055+0000", 2024-04-26T01:08:25.274 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.023710078999999998 2024-04-26T01:08:25.274 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.274 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.274 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.274 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398067+0000", 2024-04-26T01:08:25.274 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2455e-05 2024-04-26T01:08:25.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.275 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.275 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.275 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398160+0000", 2024-04-26T01:08:25.275 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2682000000000005e-05 2024-04-26T01:08:25.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.275 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.275 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.275 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398323+0000", 2024-04-26T01:08:25.275 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016276600000000001 2024-04-26T01:08:25.275 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.275 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.275 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.275 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.276 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs2 33/30 ECSubWrite(tid=21, reqid=client.4521.0:89, at_version=33'8, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:08:25.276 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.399506+0000", 2024-04-26T01:08:25.276 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 25.834102858000001, 2024-04-26T01:08:25.276 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012907333999999999, 2024-04-26T01:08:25.276 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.276 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.276 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:25.276 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.276 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.276 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.276 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.399506+0000", 2024-04-26T01:08:25.276 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.276 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.276 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.277 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.277 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.399506+0000", 2024-04-26T01:08:25.277 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.277 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.277 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.277 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.277 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.399507+0000", 2024-04-26T01:08:25.277 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0800000000000004e-07 2024-04-26T01:08:25.277 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.277 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.277 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.277 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.399525+0000", 2024-04-26T01:08:25.277 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7342000000000001e-05 2024-04-26T01:08:25.278 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.278 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.278 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.278 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411386+0000", 2024-04-26T01:08:25.278 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011861772 2024-04-26T01:08:25.278 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.278 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.278 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.278 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411395+0000", 2024-04-26T01:08:25.278 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0019999999999995e-06 2024-04-26T01:08:25.278 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.278 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.278 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.278 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411484+0000", 2024-04-26T01:08:25.279 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8993000000000007e-05 2024-04-26T01:08:25.279 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.279 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.279 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.279 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.411495+0000", 2024-04-26T01:08:25.279 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1182000000000001e-05 2024-04-26T01:08:25.279 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.279 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.279 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.279 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.412364+0000", 2024-04-26T01:08:25.279 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00086887599999999998 2024-04-26T01:08:25.279 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.279 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.279 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.280 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.412414+0000", 2024-04-26T01:08:25.280 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9459000000000002e-05 2024-04-26T01:08:25.280 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.280 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.280 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.280 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.280 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.280 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:239 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~393216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:25.280 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.342338+0000", 2024-04-26T01:08:25.280 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 21.891271569000001, 2024-04-26T01:08:25.280 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.019617285000000002, 2024-04-26T01:08:25.280 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.280 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.281 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.281 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.281 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.281 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.281 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 239 2024-04-26T01:08:25.281 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.281 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.281 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.281 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.281 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342338+0000", 2024-04-26T01:08:25.281 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.281 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.281 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.282 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.282 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342338+0000", 2024-04-26T01:08:25.282 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.282 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.282 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.282 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.282 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342340+0000", 2024-04-26T01:08:25.282 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4700000000000001e-06 2024-04-26T01:08:25.282 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.282 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.282 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.282 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342351+0000", 2024-04-26T01:08:25.282 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0925e-05 2024-04-26T01:08:25.283 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.283 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.283 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.283 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342379+0000", 2024-04-26T01:08:25.283 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8140000000000002e-05 2024-04-26T01:08:25.283 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.283 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.283 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.283 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342395+0000", 2024-04-26T01:08:25.283 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5620999999999998e-05 2024-04-26T01:08:25.283 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.283 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.283 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.284 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342550+0000", 2024-04-26T01:08:25.284 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015483000000000001 2024-04-26T01:08:25.284 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.284 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.284 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op must wait for map", 2024-04-26T01:08:25.284 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342566+0000", 2024-04-26T01:08:25.284 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6214000000000001e-05 2024-04-26T01:08:25.284 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.284 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.284 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.284 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.349020+0000", 2024-04-26T01:08:25.284 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0064544240000000003 2024-04-26T01:08:25.284 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.284 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.284 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.285 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.349117+0000", 2024-04-26T01:08:25.285 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6212000000000007e-05 2024-04-26T01:08:25.285 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.285 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.285 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.285 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.361955+0000", 2024-04-26T01:08:25.285 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012838449 2024-04-26T01:08:25.285 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.285 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.285 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.285 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.285 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.285 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:240 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:25.285 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.342427+0000", 2024-04-26T01:08:25.285 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 21.891182358999998, 2024-04-26T01:08:25.286 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.019852153000000001, 2024-04-26T01:08:25.286 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.286 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.286 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.286 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.286 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.286 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.286 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 240 2024-04-26T01:08:25.286 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.286 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.286 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.286 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.286 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342427+0000", 2024-04-26T01:08:25.286 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.287 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.287 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.287 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.287 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342427+0000", 2024-04-26T01:08:25.287 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.287 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.287 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.287 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.287 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342428+0000", 2024-04-26T01:08:25.287 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2200000000000003e-07 2024-04-26T01:08:25.287 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.287 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.287 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.287 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342435+0000", 2024-04-26T01:08:25.288 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0180000000000004e-06 2024-04-26T01:08:25.288 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.288 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.288 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.288 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342451+0000", 2024-04-26T01:08:25.288 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6011e-05 2024-04-26T01:08:25.288 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.288 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.288 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.288 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342459+0000", 2024-04-26T01:08:25.288 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1370000000000002e-06 2024-04-26T01:08:25.288 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.289 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.289 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.289 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342766+0000", 2024-04-26T01:08:25.289 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00030684000000000001 2024-04-26T01:08:25.289 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.289 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.289 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:25.289 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342784+0000", 2024-04-26T01:08:25.289 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7773000000000001e-05 2024-04-26T01:08:25.289 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.289 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.289 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.289 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.349542+0000", 2024-04-26T01:08:25.289 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0067586349999999998 2024-04-26T01:08:25.289 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.290 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.290 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.290 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.349627+0000", 2024-04-26T01:08:25.290 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4790999999999995e-05 2024-04-26T01:08:25.290 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.290 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.290 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.290 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.362279+0000", 2024-04-26T01:08:25.290 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012652126 2024-04-26T01:08:25.290 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.290 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.290 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.290 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.290 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.291 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:241 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:25.291 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.342661+0000", 2024-04-26T01:08:25.291 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 21.890948509000001, 2024-04-26T01:08:25.291 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.020164171000000002, 2024-04-26T01:08:25.291 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.291 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.291 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.291 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.291 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.291 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.291 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 241 2024-04-26T01:08:25.291 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.291 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.291 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.292 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.292 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342661+0000", 2024-04-26T01:08:25.292 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.292 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.292 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.292 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.292 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342661+0000", 2024-04-26T01:08:25.292 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.292 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.292 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.292 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.292 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342662+0000", 2024-04-26T01:08:25.292 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0100000000000003e-07 2024-04-26T01:08:25.292 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.292 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.293 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.293 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342668+0000", 2024-04-26T01:08:25.293 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2369999999999997e-06 2024-04-26T01:08:25.293 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.293 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.293 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.293 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342682+0000", 2024-04-26T01:08:25.293 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4166e-05 2024-04-26T01:08:25.293 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.293 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.293 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.293 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342691+0000", 2024-04-26T01:08:25.293 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8000000000000004e-06 2024-04-26T01:08:25.293 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.294 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.294 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.294 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342844+0000", 2024-04-26T01:08:25.294 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015316599999999999 2024-04-26T01:08:25.294 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.294 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.294 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:25.294 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342853+0000", 2024-04-26T01:08:25.294 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9700000000000005e-06 2024-04-26T01:08:25.294 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.294 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.294 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.294 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.350630+0000", 2024-04-26T01:08:25.294 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0077770890000000001 2024-04-26T01:08:25.295 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.295 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.295 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.295 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.350718+0000", 2024-04-26T01:08:25.295 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8156000000000001e-05 2024-04-26T01:08:25.295 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.295 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.295 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.295 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.362825+0000", 2024-04-26T01:08:25.295 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012106686 2024-04-26T01:08:25.295 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.295 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.295 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.295 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.295 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.296 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:276 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:25.296 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.413967+0000", 2024-04-26T01:08:25.296 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 20.819641894, 2024-04-26T01:08:25.296 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014303402, 2024-04-26T01:08:25.296 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.296 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.296 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.296 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.296 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.296 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.296 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 276 2024-04-26T01:08:25.296 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.296 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413967+0000", 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413967+0000", 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413969+0000", 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.24e-06 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.297 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413996+0000", 2024-04-26T01:08:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7093999999999998e-05 2024-04-26T01:08:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414016+0000", 2024-04-26T01:08:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0072999999999999e-05 2024-04-26T01:08:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414031+0000", 2024-04-26T01:08:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5159e-05 2024-04-26T01:08:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.299 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.299 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.299 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.420581+0000", 2024-04-26T01:08:25.299 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0065497519999999998 2024-04-26T01:08:25.299 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.299 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.299 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.299 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.420691+0000", 2024-04-26T01:08:25.299 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000110259 2024-04-26T01:08:25.299 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.299 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.299 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.299 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.428271+0000", 2024-04-26T01:08:25.299 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0075798250000000001 2024-04-26T01:08:25.299 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.300 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.300 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.300 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.300 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.300 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs2 41/30 ECSubWrite(tid=91, reqid=client.4521.0:345, at_version=41'27, trim_to=0'0, roll_forward_to=41'26))", 2024-04-26T01:08:25.300 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:07.610935+0000", 2024-04-26T01:08:25.300 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.622673985999999, 2024-04-26T01:08:25.300 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.629675231, 2024-04-26T01:08:25.300 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.300 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.300 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:25.300 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.300 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.301 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.301 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.610935+0000", 2024-04-26T01:08:25.301 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.301 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.301 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.301 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.301 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.610935+0000", 2024-04-26T01:08:25.301 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.301 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.301 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.301 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.301 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.610938+0000", 2024-04-26T01:08:25.301 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.39e-06 2024-04-26T01:08:25.302 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.302 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.302 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.302 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.611405+0000", 2024-04-26T01:08:25.302 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00046744899999999999 2024-04-26T01:08:25.302 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.302 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.302 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.302 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238064+0000", 2024-04-26T01:08:25.302 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.62665866100000001 2024-04-26T01:08:25.302 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.302 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.302 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.303 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238080+0000", 2024-04-26T01:08:25.303 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6623e-05 2024-04-26T01:08:25.303 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.303 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.303 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.303 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238214+0000", 2024-04-26T01:08:25.303 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013316900000000001 2024-04-26T01:08:25.303 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.303 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.303 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.303 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238235+0000", 2024-04-26T01:08:25.303 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0937e-05 2024-04-26T01:08:25.303 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.303 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.304 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.304 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.240510+0000", 2024-04-26T01:08:25.304 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022752860000000001 2024-04-26T01:08:25.304 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.304 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.304 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.304 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.240611+0000", 2024-04-26T01:08:25.304 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000100716 2024-04-26T01:08:25.304 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.304 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.304 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.304 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.304 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.305 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:392 3.6s0 3:61b5d9f4:::smithi16428812-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:25.305 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.463673+0000", 2024-04-26T01:08:25.305 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 15.769935856, 2024-04-26T01:08:25.305 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010240143, 2024-04-26T01:08:25.305 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.305 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.305 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.305 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.305 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.305 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.305 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 392 2024-04-26T01:08:25.305 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.305 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.305 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.306 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.306 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463673+0000", 2024-04-26T01:08:25.306 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.306 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.306 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.306 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.306 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463673+0000", 2024-04-26T01:08:25.306 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.306 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.306 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.306 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.306 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463674+0000", 2024-04-26T01:08:25.306 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.9899999999999999e-07 2024-04-26T01:08:25.307 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.307 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.307 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.307 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463677+0000", 2024-04-26T01:08:25.307 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0460000000000001e-06 2024-04-26T01:08:25.307 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.307 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.307 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.307 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463686+0000", 2024-04-26T01:08:25.307 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1290000000000002e-06 2024-04-26T01:08:25.307 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.307 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.307 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.307 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.463694+0000", 2024-04-26T01:08:25.308 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.4919999999999996e-06 2024-04-26T01:08:25.308 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.308 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.308 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.308 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.466138+0000", 2024-04-26T01:08:25.308 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024443120000000001 2024-04-26T01:08:25.308 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.308 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.308 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:25.308 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.466255+0000", 2024-04-26T01:08:25.308 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000116246 2024-04-26T01:08:25.308 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.308 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.308 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.309 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.472474+0000", 2024-04-26T01:08:25.309 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0062197420000000003 2024-04-26T01:08:25.309 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.309 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.309 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.309 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.472536+0000", 2024-04-26T01:08:25.309 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2008000000000007e-05 2024-04-26T01:08:25.309 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.309 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.309 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.309 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.473914+0000", 2024-04-26T01:08:25.309 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001377369 2024-04-26T01:08:25.309 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.309 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.310 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.310 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.310 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.310 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 43/30 ECSubWriteReply(tid=33, last_complete=43'17, committed=1, applied=1))", 2024-04-26T01:08:25.310 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.473084+0000", 2024-04-26T01:08:25.310 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 15.760525743000001, 2024-04-26T01:08:25.310 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.913281966, 2024-04-26T01:08:25.310 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.310 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.310 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:25.310 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.310 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.310 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.311 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.473084+0000", 2024-04-26T01:08:25.311 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.311 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.311 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.311 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.311 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.473084+0000", 2024-04-26T01:08:25.311 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.311 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.311 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.311 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.311 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.473084+0000", 2024-04-26T01:08:25.311 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6300000000000005e-07 2024-04-26T01:08:25.311 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.473088+0000", 2024-04-26T01:08:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7620000000000001e-06 2024-04-26T01:08:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.386034+0000", 2024-04-26T01:08:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.91294643399999997 2024-04-26T01:08:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.386055+0000", 2024-04-26T01:08:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0492999999999998e-05 2024-04-26T01:08:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.313 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.313 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.313 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.386234+0000", 2024-04-26T01:08:25.313 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000179259 2024-04-26T01:08:25.313 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.313 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.313 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.313 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.386366+0000", 2024-04-26T01:08:25.313 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013135499999999999 2024-04-26T01:08:25.313 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.313 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.313 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.313 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.313 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.314 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:592 3.6s0 3:69323dd0:::smithi16428812-38:head [read 0~1884160,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:25.314 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.164665+0000", 2024-04-26T01:08:25.314 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.068944745, 2024-04-26T01:08:25.314 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013531341000000001, 2024-04-26T01:08:25.314 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.314 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.314 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.314 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.314 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.314 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.314 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 592 2024-04-26T01:08:25.314 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.314 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.314 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.315 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.315 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164665+0000", 2024-04-26T01:08:25.315 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.315 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.315 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.315 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.315 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164665+0000", 2024-04-26T01:08:25.315 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.315 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.315 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.315 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.315 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164667+0000", 2024-04-26T01:08:25.315 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5660000000000001e-06 2024-04-26T01:08:25.315 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.316 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.316 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.316 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164676+0000", 2024-04-26T01:08:25.316 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8400000000000001e-06 2024-04-26T01:08:25.316 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.316 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.316 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.316 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164701+0000", 2024-04-26T01:08:25.316 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4924000000000001e-05 2024-04-26T01:08:25.316 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.316 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.316 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.316 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164714+0000", 2024-04-26T01:08:25.316 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3539e-05 2024-04-26T01:08:25.317 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.317 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.317 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.317 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164850+0000", 2024-04-26T01:08:25.317 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013509700000000001 2024-04-26T01:08:25.317 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.317 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.317 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.317 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.165070+0000", 2024-04-26T01:08:25.317 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00022075 2024-04-26T01:08:25.317 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.317 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.317 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.317 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.178196+0000", 2024-04-26T01:08:25.317 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013125625 2024-04-26T01:08:25.318 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.318 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.318 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.318 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.318 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.318 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:593 3.6s0 3:69323dd0:::smithi16428812-38:head [read 0~1884160,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:25.318 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.164747+0000", 2024-04-26T01:08:25.318 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.0688626919999999, 2024-04-26T01:08:25.318 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014725004, 2024-04-26T01:08:25.318 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.318 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.318 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.318 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.318 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.319 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.319 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 593 2024-04-26T01:08:25.319 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.319 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.319 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.319 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.319 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164747+0000", 2024-04-26T01:08:25.319 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.319 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.319 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.319 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.319 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164747+0000", 2024-04-26T01:08:25.319 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.319 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.319 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.320 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.320 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164748+0000", 2024-04-26T01:08:25.320 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0410000000000001e-06 2024-04-26T01:08:25.320 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.320 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.320 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.320 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164754+0000", 2024-04-26T01:08:25.320 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3929999999999996e-06 2024-04-26T01:08:25.320 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.320 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.320 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.320 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164769+0000", 2024-04-26T01:08:25.320 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4834999999999999e-05 2024-04-26T01:08:25.320 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.321 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.321 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.321 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164779+0000", 2024-04-26T01:08:25.321 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0117e-05 2024-04-26T01:08:25.321 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.321 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.321 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.321 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.165879+0000", 2024-04-26T01:08:25.321 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0010995899999999999 2024-04-26T01:08:25.321 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.321 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.321 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.321 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.166031+0000", 2024-04-26T01:08:25.321 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015242399999999999 2024-04-26T01:08:25.322 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.322 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.322 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.322 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.179472+0000", 2024-04-26T01:08:25.322 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013440604 2024-04-26T01:08:25.322 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.322 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.322 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.322 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.322 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.322 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:594 3.6s0 3:69323dd0:::smithi16428812-38:head [read 0~1884160,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:25.322 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.164820+0000", 2024-04-26T01:08:25.322 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.0687891190000001, 2024-04-26T01:08:25.322 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.016408565, 2024-04-26T01:08:25.323 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.323 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.323 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.323 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.323 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.323 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.323 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 594 2024-04-26T01:08:25.323 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.323 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.323 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.323 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.323 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164820+0000", 2024-04-26T01:08:25.323 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.324 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.324 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.324 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.324 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164820+0000", 2024-04-26T01:08:25.324 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.324 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.324 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.324 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.324 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164821+0000", 2024-04-26T01:08:25.324 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2099999999999995e-07 2024-04-26T01:08:25.324 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.324 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.324 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.324 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164826+0000", 2024-04-26T01:08:25.324 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4430000000000002e-06 2024-04-26T01:08:25.325 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.325 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.325 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.325 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164839+0000", 2024-04-26T01:08:25.325 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2622999999999999e-05 2024-04-26T01:08:25.325 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.325 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.325 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.325 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164851+0000", 2024-04-26T01:08:25.325 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1497e-05 2024-04-26T01:08:25.325 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.325 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.325 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.325 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.166481+0000", 2024-04-26T01:08:25.326 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001630432 2024-04-26T01:08:25.326 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.326 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.326 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.326 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.166572+0000", 2024-04-26T01:08:25.326 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1268999999999996e-05 2024-04-26T01:08:25.326 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.326 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.326 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.326 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.181229+0000", 2024-04-26T01:08:25.326 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.01465648 2024-04-26T01:08:25.326 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.326 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.326 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.327 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.327 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.327 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/30 ECSubWrite(tid=318, reqid=client.4521.0:601, at_version=58'41, trim_to=0'0, roll_forward_to=58'39))", 2024-04-26T01:08:25.327 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.359815+0000", 2024-04-26T01:08:25.327 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.87379469899999995, 2024-04-26T01:08:25.327 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.11192218399999999, 2024-04-26T01:08:25.327 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.327 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.327 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:25.327 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.327 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.327 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.328 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.359815+0000", 2024-04-26T01:08:25.328 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.328 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.328 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.328 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.328 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.359815+0000", 2024-04-26T01:08:25.328 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.328 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.328 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.328 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.328 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.359816+0000", 2024-04-26T01:08:25.328 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1429999999999999e-06 2024-04-26T01:08:25.328 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.328 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.328 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.329 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.359875+0000", 2024-04-26T01:08:25.329 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9364000000000003e-05 2024-04-26T01:08:25.329 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.329 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.329 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.329 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.470037+0000", 2024-04-26T01:08:25.329 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.110162076 2024-04-26T01:08:25.329 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.329 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.329 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.329 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.470055+0000", 2024-04-26T01:08:25.329 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7863000000000002e-05 2024-04-26T01:08:25.329 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.330 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.330 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.330 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.470147+0000", 2024-04-26T01:08:25.330 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2103999999999998e-05 2024-04-26T01:08:25.330 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.330 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.330 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.330 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.470157+0000", 2024-04-26T01:08:25.330 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0182e-05 2024-04-26T01:08:25.330 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.330 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.330 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.330 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.471667+0000", 2024-04-26T01:08:25.331 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015096739999999999 2024-04-26T01:08:25.331 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.331 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.331 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.331 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.471737+0000", 2024-04-26T01:08:25.331 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9777999999999995e-05 2024-04-26T01:08:25.331 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.331 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.331 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.331 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.331 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.331 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:25.333 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:08:25.333 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:25.331+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:25.392 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:08:25.398 INFO:tasks.rados.rados.0.smithi164.stdout:234: finishing copy_from to smithi16428812-40 2024-04-26T01:08:25.398 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 40 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-26T01:08:25.399 INFO:tasks.rados.rados.0.smithi164.stdout:234: done (4 left) 2024-04-26T01:08:25.399 INFO:tasks.rados.rados.0.smithi164.stdout:235: done (3 left) 2024-04-26T01:08:25.399 INFO:tasks.rados.rados.0.smithi164.stdout:236: done (2 left) 2024-04-26T01:08:25.399 INFO:tasks.rados.rados.0.smithi164.stdout:237: done (1 left) 2024-04-26T01:08:25.399 INFO:tasks.rados.rados.0.smithi164.stdout:240: rollback oid 29 current snap is 14 2024-04-26T01:08:25.399 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 29 to 13 2024-04-26T01:08:25.399 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 44 current snap is 14 2024-04-26T01:08:25.399 INFO:tasks.rados.rados.0.smithi164.stdout:241: seq_num 113 ranges {3866624=737280,4603904=557056,5160960=606208} 2024-04-26T01:08:25.434 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:25.432+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:25.459 INFO:tasks.rados.rados.0.smithi164.stdout:241: writing smithi16428812-44 from 3866624 to 4603904 tid 1 2024-04-26T01:08:25.465 INFO:tasks.rados.rados.0.smithi164.stdout:241: writing smithi16428812-44 from 4603904 to 5160960 tid 2 2024-04-26T01:08:25.470 INFO:tasks.rados.rados.0.smithi164.stdout:241: writing smithi16428812-44 from 5160960 to 5767168 tid 3 2024-04-26T01:08:25.471 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 25 current snap is 14 2024-04-26T01:08:25.471 INFO:tasks.rados.rados.0.smithi164.stdout:240: finishing rollback tid 0 to smithi16428812-29 2024-04-26T01:08:25.471 INFO:tasks.rados.rados.0.smithi164.stdout:242: seq_num 114 ranges {0=753664,753664=688128,1441792=475136} 2024-04-26T01:08:25.473 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:25.473 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:25.473 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.473 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:618 3.bs0 3:d071ab05:::smithi16428812-44:head [append 0~606208 in=606208b] snapc f=[f,d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-26T01:08:25.473 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:25.471261+0000", 2024-04-26T01:08:25.473 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0012431199999999999, 2024-04-26T01:08:25.473 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001268922, 2024-04-26T01:08:25.473 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.473 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.473 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.473 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.473 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.474 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.474 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 618 2024-04-26T01:08:25.474 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.474 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.474 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.474 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.474 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.471261+0000", 2024-04-26T01:08:25.474 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.474 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.474 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.474 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.474 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.471261+0000", 2024-04-26T01:08:25.474 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.475 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.475 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.475 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.475 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.471263+0000", 2024-04-26T01:08:25.475 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9350000000000001e-06 2024-04-26T01:08:25.475 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.475 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.475 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.475 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.471958+0000", 2024-04-26T01:08:25.475 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000695339 2024-04-26T01:08:25.475 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.475 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.475 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.475 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.471986+0000", 2024-04-26T01:08:25.475 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7370999999999999e-05 2024-04-26T01:08:25.476 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.476 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.476 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.476 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.472003+0000", 2024-04-26T01:08:25.476 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7323e-05 2024-04-26T01:08:25.476 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.476 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.476 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.476 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.472365+0000", 2024-04-26T01:08:25.476 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00036169199999999999 2024-04-26T01:08:25.476 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.476 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.476 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.476 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.472473+0000", 2024-04-26T01:08:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000108041 2024-04-26T01:08:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:619 3.bs0 3.a0d58e0b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-26T01:08:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:25.472038+0000", 2024-04-26T01:08:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.00046619499999999998, 2024-04-26T01:08:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00057414200000000003, 2024-04-26T01:08:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-04-26T01:08:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 619 2024-04-26T01:08:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.472038+0000", 2024-04-26T01:08:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.472038+0000", 2024-04-26T01:08:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.479 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.479 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.472040+0000", 2024-04-26T01:08:25.479 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.626e-06 2024-04-26T01:08:25.479 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.479 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.479 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.479 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.472053+0000", 2024-04-26T01:08:25.479 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3057000000000001e-05 2024-04-26T01:08:25.479 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.479 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.479 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.479 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.472075+0000", 2024-04-26T01:08:25.479 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2090999999999999e-05 2024-04-26T01:08:25.479 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.479 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.480 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.480 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.472091+0000", 2024-04-26T01:08:25.480 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5763000000000001e-05 2024-04-26T01:08:25.480 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.480 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.480 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.480 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.480 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.480 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:620 3.bs0 3.a0d58e0b (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-04-26T01:08:25.480 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:25.472110+0000", 2024-04-26T01:08:25.480 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.00039447899999999999, 2024-04-26T01:08:25.480 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00056749600000000002, 2024-04-26T01:08:25.480 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.481 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.481 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-04-26T01:08:25.481 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.481 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.481 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.481 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 620 2024-04-26T01:08:25.481 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.481 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.481 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.481 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.481 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.472110+0000", 2024-04-26T01:08:25.481 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.482 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.482 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.482 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.482 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.472110+0000", 2024-04-26T01:08:25.482 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.482 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.482 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.482 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.482 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.472111+0000", 2024-04-26T01:08:25.482 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9500000000000001e-07 2024-04-26T01:08:25.482 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.482 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.483 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.483 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.472115+0000", 2024-04-26T01:08:25.483 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4510000000000002e-06 2024-04-26T01:08:25.483 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.483 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.483 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.483 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.472126+0000", 2024-04-26T01:08:25.483 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1103e-05 2024-04-26T01:08:25.483 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.483 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.483 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.483 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.472134+0000", 2024-04-26T01:08:25.483 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0509999999999997e-06 2024-04-26T01:08:25.483 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.483 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.484 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.484 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.484 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-26T01:08:25.484 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 3 2024-04-26T01:08:25.484 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:25.484 INFO:tasks.rados.rados.0.smithi164.stdout:242: writing smithi16428812-25 from 0 to 753664 tid 1 2024-04-26T01:08:25.484 INFO:tasks.rados.rados.0.smithi164.stdout:242: writing smithi16428812-25 from 753664 to 1441792 tid 2 2024-04-26T01:08:25.488 INFO:tasks.rados.rados.0.smithi164.stdout:242: writing smithi16428812-25 from 1441792 to 1916928 tid 3 2024-04-26T01:08:25.488 INFO:tasks.rados.rados.0.smithi164.stdout:243: read oid 42 snap -1 2024-04-26T01:08:25.488 INFO:tasks.rados.rados.0.smithi164.stdout:243: expect (ObjNum 91 snap 7 seq_num 91) 2024-04-26T01:08:25.488 INFO:tasks.rados.rados.0.smithi164.stdout:244: read oid 9 snap -1 2024-04-26T01:08:25.488 INFO:tasks.rados.rados.0.smithi164.stdout:244: expect (ObjNum 99 snap 11 seq_num 99) 2024-04-26T01:08:25.488 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 23 current snap is 14 2024-04-26T01:08:25.488 INFO:tasks.rados.rados.0.smithi164.stdout:245: seq_num 115 ranges {376832=770048,1146880=704512,1851392=458752} 2024-04-26T01:08:25.494 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:08:25.498 INFO:tasks.rados.rados.0.smithi164.stdout:245: writing smithi16428812-23 from 376832 to 1146880 tid 1 2024-04-26T01:08:25.505 INFO:tasks.rados.rados.0.smithi164.stdout:245: writing smithi16428812-23 from 1146880 to 1851392 tid 2 2024-04-26T01:08:25.509 INFO:tasks.rados.rados.0.smithi164.stdout:245: writing smithi16428812-23 from 1851392 to 2310144 tid 3 2024-04-26T01:08:25.509 INFO:tasks.rados.rados.0.smithi164.stdout:246: snap_remove snap 11 2024-04-26T01:08:25.509 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 30 v 51 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-26T01:08:25.509 INFO:tasks.rados.rados.0.smithi164.stdout:240: finishing rollback tid 1 to smithi16428812-29 2024-04-26T01:08:25.535 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:25.535 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:25.535 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:08:25.535 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:25.535 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:25.536+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:25.547 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed 2024-04-26T01:08:25.636 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:25.636+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:25.665 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:25.665 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:25.665 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-04-26T01:08:25.665 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-04-26T01:08:25.665 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:25.681 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:08:25.739 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:25.740+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:25.805 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:25.805 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:25.805 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:25.805 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:25.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.806 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:3 3.1s0 3:8bb39a4d:::smithi16428812-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:25.806 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.300507+0000", 2024-04-26T01:08:25.806 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 26.500851520000001, 2024-04-26T01:08:25.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.030795241000000001, 2024-04-26T01:08:25.806 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.806 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.806 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.806 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.806 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.806 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.806 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 3 2024-04-26T01:08:25.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.806 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.807 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.807 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300507+0000", 2024-04-26T01:08:25.807 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.807 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.807 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.807 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300507+0000", 2024-04-26T01:08:25.807 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.807 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.807 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.807 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300507+0000", 2024-04-26T01:08:25.808 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2600000000000002e-07 2024-04-26T01:08:25.808 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.808 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.808 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.808 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300510+0000", 2024-04-26T01:08:25.808 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1940000000000001e-06 2024-04-26T01:08:25.808 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.808 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.808 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.808 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300524+0000", 2024-04-26T01:08:25.808 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4663e-05 2024-04-26T01:08:25.808 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.808 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.808 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.809 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.300529+0000", 2024-04-26T01:08:25.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8919999999999999e-06 2024-04-26T01:08:25.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.809 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.809 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303652+0000", 2024-04-26T01:08:25.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0031224500000000001 2024-04-26T01:08:25.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.809 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:25.809 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.303744+0000", 2024-04-26T01:08:25.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2109999999999997e-05 2024-04-26T01:08:25.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.810 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.810 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319709+0000", 2024-04-26T01:08:25.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015965231 2024-04-26T01:08:25.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.810 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.810 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.319786+0000", 2024-04-26T01:08:25.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7449999999999999e-05 2024-04-26T01:08:25.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.810 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.810 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.331302+0000", 2024-04-26T01:08:25.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011515724999999999 2024-04-26T01:08:25.811 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.811 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.811 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.811 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.811 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.811 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:38 3.7s0 3:ed7aa414:::smithi16428812-13:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:25.811 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.321672+0000", 2024-04-26T01:08:25.811 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 26.479686253000001, 2024-04-26T01:08:25.811 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.035147350000000001, 2024-04-26T01:08:25.811 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.811 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.811 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:25.811 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.811 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.812 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.812 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 38 2024-04-26T01:08:25.812 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.812 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.812 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.812 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.812 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321672+0000", 2024-04-26T01:08:25.812 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.812 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.812 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.812 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.812 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321672+0000", 2024-04-26T01:08:25.812 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.812 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.813 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.813 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321673+0000", 2024-04-26T01:08:25.813 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0100000000000001e-06 2024-04-26T01:08:25.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.813 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.813 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321680+0000", 2024-04-26T01:08:25.813 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.024e-06 2024-04-26T01:08:25.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.813 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.813 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321698+0000", 2024-04-26T01:08:25.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8158000000000001e-05 2024-04-26T01:08:25.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321707+0000", 2024-04-26T01:08:25.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6419999999999996e-06 2024-04-26T01:08:25.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325010+0000", 2024-04-26T01:08:25.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033028720000000001 2024-04-26T01:08:25.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325101+0000", 2024-04-26T01:08:25.815 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1613999999999994e-05 2024-04-26T01:08:25.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325552+0000", 2024-04-26T01:08:25.815 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00045083800000000001 2024-04-26T01:08:25.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.328138+0000", 2024-04-26T01:08:25.816 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0025861159999999998 2024-04-26T01:08:25.816 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.816 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.816 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:25.816 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.356713+0000", 2024-04-26T01:08:25.816 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.028574609000000001 2024-04-26T01:08:25.816 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.816 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.816 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.816 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.356819+0000", 2024-04-26T01:08:25.816 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000106467 2024-04-26T01:08:25.816 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.816 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.817 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.817 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:39 3.7s0 3:ed7aa414:::smithi16428812-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:25.817 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.321721+0000", 2024-04-26T01:08:25.817 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 26.479637324999999, 2024-04-26T01:08:25.817 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.038875445000000002, 2024-04-26T01:08:25.817 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.817 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.817 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.817 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.817 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.817 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.818 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 39 2024-04-26T01:08:25.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.818 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321721+0000", 2024-04-26T01:08:25.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321721+0000", 2024-04-26T01:08:25.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.819 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321722+0000", 2024-04-26T01:08:25.819 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2999999999999999e-07 2024-04-26T01:08:25.819 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.819 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321726+0000", 2024-04-26T01:08:25.819 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7639999999999999e-06 2024-04-26T01:08:25.819 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.819 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321736+0000", 2024-04-26T01:08:25.820 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0473e-05 2024-04-26T01:08:25.820 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.820 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.820 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.820 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.321747+0000", 2024-04-26T01:08:25.820 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0981e-05 2024-04-26T01:08:25.820 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.820 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.820 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.820 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.326352+0000", 2024-04-26T01:08:25.820 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046046239999999999 2024-04-26T01:08:25.820 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.820 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.821 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:25.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.326448+0000", 2024-04-26T01:08:25.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6668000000000003e-05 2024-04-26T01:08:25.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.821 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.356889+0000", 2024-04-26T01:08:25.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.030441071 2024-04-26T01:08:25.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.821 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.357378+0000", 2024-04-26T01:08:25.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00048859800000000003 2024-04-26T01:08:25.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.360597+0000", 2024-04-26T01:08:25.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032184359999999999 2024-04-26T01:08:25.822 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.822 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.822 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.822 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.822 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:65 3.7s0 3:e163c084:::smithi16428812-22:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:25.822 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.364410+0000", 2024-04-26T01:08:25.822 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 26.436948709999999, 2024-04-26T01:08:25.823 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.081696361999999995, 2024-04-26T01:08:25.823 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.823 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.823 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:25.823 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.823 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.823 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.823 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 65 2024-04-26T01:08:25.823 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.823 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.823 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.823 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.823 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364410+0000", 2024-04-26T01:08:25.823 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.824 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.824 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.824 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.824 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364410+0000", 2024-04-26T01:08:25.824 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.824 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.824 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.824 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.824 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364411+0000", 2024-04-26T01:08:25.824 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.102e-06 2024-04-26T01:08:25.824 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.824 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.824 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.825 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364416+0000", 2024-04-26T01:08:25.825 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8389999999999997e-06 2024-04-26T01:08:25.825 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.825 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.825 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.825 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364424+0000", 2024-04-26T01:08:25.825 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6100000000000006e-06 2024-04-26T01:08:25.825 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.825 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.825 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.825 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364432+0000", 2024-04-26T01:08:25.825 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5320000000000001e-06 2024-04-26T01:08:25.825 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.825 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.825 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.826 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.365925+0000", 2024-04-26T01:08:25.826 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001493592 2024-04-26T01:08:25.826 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.826 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.826 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.826 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366035+0000", 2024-04-26T01:08:25.826 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000109448 2024-04-26T01:08:25.826 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.826 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.826 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.826 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366490+0000", 2024-04-26T01:08:25.826 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000455427 2024-04-26T01:08:25.827 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.827 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.827 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.827 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.367513+0000", 2024-04-26T01:08:25.827 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001023014 2024-04-26T01:08:25.827 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.827 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.827 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:25.827 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446008+0000", 2024-04-26T01:08:25.827 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.078494871999999993 2024-04-26T01:08:25.827 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.827 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.827 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.827 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446106+0000", 2024-04-26T01:08:25.828 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7925999999999997e-05 2024-04-26T01:08:25.828 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.828 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.828 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.828 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.828 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.828 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:66 3.7s0 3:e163c084:::smithi16428812-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:25.828 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.364442+0000", 2024-04-26T01:08:25.828 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 26.436916182000001, 2024-04-26T01:08:25.828 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.088465429999999998, 2024-04-26T01:08:25.828 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.828 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.828 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.828 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.829 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.829 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.829 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 66 2024-04-26T01:08:25.829 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.829 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.829 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.829 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.829 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364442+0000", 2024-04-26T01:08:25.829 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.829 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.829 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.829 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.829 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364442+0000", 2024-04-26T01:08:25.829 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.830 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.830 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.830 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.830 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364443+0000", 2024-04-26T01:08:25.830 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7899999999999998e-07 2024-04-26T01:08:25.830 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.830 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.830 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.830 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364446+0000", 2024-04-26T01:08:25.830 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9119999999999998e-06 2024-04-26T01:08:25.830 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.830 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.830 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.831 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364454+0000", 2024-04-26T01:08:25.831 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0229999999999992e-06 2024-04-26T01:08:25.831 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.831 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.831 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.831 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364461+0000", 2024-04-26T01:08:25.831 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0110000000000003e-06 2024-04-26T01:08:25.831 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.831 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.831 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.831 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.367245+0000", 2024-04-26T01:08:25.831 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027847499999999999 2024-04-26T01:08:25.831 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.831 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.832 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:25.832 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.367332+0000", 2024-04-26T01:08:25.832 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6118000000000005e-05 2024-04-26T01:08:25.832 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.832 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.832 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.832 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446632+0000", 2024-04-26T01:08:25.832 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.079300689999999993 2024-04-26T01:08:25.832 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.832 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.832 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.832 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446716+0000", 2024-04-26T01:08:25.832 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3391000000000002e-05 2024-04-26T01:08:25.832 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.833 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.833 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.833 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.452908+0000", 2024-04-26T01:08:25.833 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0061919560000000002 2024-04-26T01:08:25.833 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.833 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.833 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.833 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.833 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.833 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:102 3.1s0 3:88871701:::smithi16428812-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:25.833 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.413770+0000", 2024-04-26T01:08:25.833 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 26.387588487999999, 2024-04-26T01:08:25.833 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.035191759000000003, 2024-04-26T01:08:25.833 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.834 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.834 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.834 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.834 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.834 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.834 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 102 2024-04-26T01:08:25.834 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.834 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.834 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.834 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.834 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413770+0000", 2024-04-26T01:08:25.834 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.835 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.835 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.835 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.835 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413770+0000", 2024-04-26T01:08:25.835 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.835 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.835 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.835 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.835 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413770+0000", 2024-04-26T01:08:25.835 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4799999999999998e-07 2024-04-26T01:08:25.835 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.835 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.835 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.835 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413773+0000", 2024-04-26T01:08:25.836 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7499999999999999e-06 2024-04-26T01:08:25.836 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.836 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.836 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.836 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413782+0000", 2024-04-26T01:08:25.836 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7129999999999994e-06 2024-04-26T01:08:25.836 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.836 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.836 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.836 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.413791+0000", 2024-04-26T01:08:25.836 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4280000000000002e-06 2024-04-26T01:08:25.836 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.836 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.837 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.837 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.416262+0000", 2024-04-26T01:08:25.837 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024706210000000001 2024-04-26T01:08:25.837 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.837 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.837 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:25.837 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.416358+0000", 2024-04-26T01:08:25.837 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.5775999999999999e-05 2024-04-26T01:08:25.837 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.837 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.837 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.837 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.419853+0000", 2024-04-26T01:08:25.837 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0034948790000000002 2024-04-26T01:08:25.837 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.838 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.838 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.838 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.419904+0000", 2024-04-26T01:08:25.838 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1206999999999999e-05 2024-04-26T01:08:25.838 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.838 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.838 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.838 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.448962+0000", 2024-04-26T01:08:25.838 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.029057837 2024-04-26T01:08:25.838 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.838 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.838 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.838 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.838 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.839 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:145 3.bs0 3:d8d6d69b:::smithi16428812-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:25.839 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.477949+0000", 2024-04-26T01:08:25.839 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 26.323409514000002, 2024-04-26T01:08:25.839 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.030723740999999999, 2024-04-26T01:08:25.839 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.839 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.839 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.839 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.839 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.839 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.839 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 145 2024-04-26T01:08:25.839 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.839 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.840 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.840 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.840 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477949+0000", 2024-04-26T01:08:25.840 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.840 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.840 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.840 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.840 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477949+0000", 2024-04-26T01:08:25.840 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.840 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.840 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.840 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.840 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477949+0000", 2024-04-26T01:08:25.840 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.5999999999999999e-07 2024-04-26T01:08:25.841 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.841 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.841 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.841 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477951+0000", 2024-04-26T01:08:25.841 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9929999999999998e-06 2024-04-26T01:08:25.841 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.841 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.841 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.841 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477955+0000", 2024-04-26T01:08:25.841 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2760000000000002e-06 2024-04-26T01:08:25.841 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.841 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.841 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.841 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.477959+0000", 2024-04-26T01:08:25.842 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7000000000000002e-06 2024-04-26T01:08:25.842 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.842 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.842 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.842 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.480261+0000", 2024-04-26T01:08:25.842 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0023015930000000002 2024-04-26T01:08:25.842 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.842 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.842 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:25.842 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.480344+0000", 2024-04-26T01:08:25.842 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3113000000000002e-05 2024-04-26T01:08:25.842 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.842 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.842 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.843 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.501225+0000", 2024-04-26T01:08:25.843 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.020881591000000001 2024-04-26T01:08:25.843 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.843 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.843 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.843 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.501312+0000", 2024-04-26T01:08:25.843 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6468e-05 2024-04-26T01:08:25.843 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.843 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.843 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.843 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.508673+0000", 2024-04-26T01:08:25.843 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0073606469999999997 2024-04-26T01:08:25.843 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.843 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.843 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.844 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.844 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.844 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.5s2 37/30 ECSubWrite(tid=22, reqid=client.4521.0:251, at_version=37'17, trim_to=0'0, roll_forward_to=36'13))", 2024-04-26T01:08:25.844 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.434740+0000", 2024-04-26T01:08:25.844 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 22.366618193000001, 2024-04-26T01:08:25.844 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.47257335099999997, 2024-04-26T01:08:25.844 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.844 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.844 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:25.844 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.844 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.844 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.844 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434740+0000", 2024-04-26T01:08:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434740+0000", 2024-04-26T01:08:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434741+0000", 2024-04-26T01:08:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8800000000000002e-07 2024-04-26T01:08:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434764+0000", 2024-04-26T01:08:25.846 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3499999999999999e-05 2024-04-26T01:08:25.846 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.846 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.846 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.846 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906023+0000", 2024-04-26T01:08:25.846 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.47125867799999999 2024-04-26T01:08:25.846 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.846 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.846 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.846 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906041+0000", 2024-04-26T01:08:25.846 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7887e-05 2024-04-26T01:08:25.846 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.846 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.846 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.847 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906172+0000", 2024-04-26T01:08:25.847 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013065200000000001 2024-04-26T01:08:25.847 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.847 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.847 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.847 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906192+0000", 2024-04-26T01:08:25.847 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0363000000000001e-05 2024-04-26T01:08:25.847 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.847 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.847 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.847 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.907240+0000", 2024-04-26T01:08:25.847 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0010479 2024-04-26T01:08:25.847 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.847 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.848 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.848 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.907314+0000", 2024-04-26T01:08:25.848 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3583000000000003e-05 2024-04-26T01:08:25.848 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.848 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.848 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.848 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.848 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.848 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:267 3.1s0 3:8f91d505:::smithi16428812-26:head [copy-from ver 9] snapc 4=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:25.848 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.413377+0000", 2024-04-26T01:08:25.848 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 21.387981299, 2024-04-26T01:08:25.848 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.033076248000000003, 2024-04-26T01:08:25.848 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.849 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.849 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:25.849 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.849 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.849 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.849 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 267 2024-04-26T01:08:25.849 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.849 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.849 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.849 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.849 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413377+0000", 2024-04-26T01:08:25.849 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.849 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.849 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.850 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.850 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413377+0000", 2024-04-26T01:08:25.850 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.850 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.850 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.850 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.850 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413382+0000", 2024-04-26T01:08:25.850 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5499999999999996e-06 2024-04-26T01:08:25.850 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.850 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.850 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.850 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413400+0000", 2024-04-26T01:08:25.850 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8646000000000001e-05 2024-04-26T01:08:25.850 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.851 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.851 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.851 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413453+0000", 2024-04-26T01:08:25.851 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2559999999999998e-05 2024-04-26T01:08:25.851 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.851 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.851 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.851 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413479+0000", 2024-04-26T01:08:25.851 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6210000000000001e-05 2024-04-26T01:08:25.851 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.851 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.851 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.851 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413657+0000", 2024-04-26T01:08:25.851 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017758800000000001 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op must wait for map", 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413721+0000", 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4286000000000001e-05 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.420637+0000", 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0069159130000000001 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.420830+0000", 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000193492 2024-04-26T01:08:25.852 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.439350+0000", 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018519566000000001 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.442330+0000", 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0029803 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.446325+0000", 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0039946699999999996 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.853 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.854 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.854 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.446453+0000", 2024-04-26T01:08:25.854 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000128467 2024-04-26T01:08:25.854 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.854 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.854 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.854 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.854 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.854 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 50/44 ECSubWriteReply(tid=226, last_complete=50'42, committed=1, applied=1))", 2024-04-26T01:08:25.854 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:16.093834+0000", 2024-04-26T01:08:25.854 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 9.7075247569999998, 2024-04-26T01:08:25.854 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.069647735000000002, 2024-04-26T01:08:25.854 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.855 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.855 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:25.855 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.855 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.855 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.855 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.093834+0000", 2024-04-26T01:08:25.855 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.855 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.855 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.855 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.855 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.093834+0000", 2024-04-26T01:08:25.855 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.855 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.855 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.855 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.856 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.093834+0000", 2024-04-26T01:08:25.856 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9800000000000003e-07 2024-04-26T01:08:25.856 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.856 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.856 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.856 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.093838+0000", 2024-04-26T01:08:25.856 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8369999999999999e-06 2024-04-26T01:08:25.856 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.856 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.856 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.856 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.163332+0000", 2024-04-26T01:08:25.856 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.069493965000000005 2024-04-26T01:08:25.856 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.856 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.857 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.857 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.163346+0000", 2024-04-26T01:08:25.857 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.341e-05 2024-04-26T01:08:25.857 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.857 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.857 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.857 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.163447+0000", 2024-04-26T01:08:25.857 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000101859 2024-04-26T01:08:25.857 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.857 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.857 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.857 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.163481+0000", 2024-04-26T01:08:25.857 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3966e-05 2024-04-26T01:08:25.857 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.857 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.858 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.858 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.858 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.858 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 52/30 ECSubWriteReply(tid=246, last_complete=52'26, committed=1, applied=1))", 2024-04-26T01:08:25.858 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:18.467431+0000", 2024-04-26T01:08:25.858 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 7.3339272710000003, 2024-04-26T01:08:25.858 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.59825841400000002, 2024-04-26T01:08:25.858 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.858 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.858 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:25.858 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.858 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.858 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.858 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:18.467431+0000", 2024-04-26T01:08:25.859 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.859 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.859 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.859 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.859 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:18.467431+0000", 2024-04-26T01:08:25.859 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.859 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.859 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.859 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.859 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:18.467432+0000", 2024-04-26T01:08:25.859 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5400000000000001e-07 2024-04-26T01:08:25.859 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.859 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.860 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.860 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:18.467437+0000", 2024-04-26T01:08:25.860 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1200000000000001e-06 2024-04-26T01:08:25.860 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.860 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.860 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.860 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.065059+0000", 2024-04-26T01:08:25.860 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.59762205800000001 2024-04-26T01:08:25.860 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.860 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.860 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.860 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.065068+0000", 2024-04-26T01:08:25.860 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9430000000000006e-06 2024-04-26T01:08:25.860 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.861 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.861 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.861 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.065136+0000", 2024-04-26T01:08:25.861 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7841000000000003e-05 2024-04-26T01:08:25.861 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.861 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.861 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.861 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.065690+0000", 2024-04-26T01:08:25.861 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00055379799999999999 2024-04-26T01:08:25.861 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.861 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.861 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.861 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.861 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.862 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 52/30 ECSubWriteReply(tid=247, last_complete=52'26, committed=1, applied=1))", 2024-04-26T01:08:25.862 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:19.067196+0000", 2024-04-26T01:08:25.862 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 6.7341623159999999, 2024-04-26T01:08:25.862 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.12773290200000001, 2024-04-26T01:08:25.862 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.862 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.862 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:25.862 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.862 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.862 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.862 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.067196+0000", 2024-04-26T01:08:25.862 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.862 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.863 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.863 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.863 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.067196+0000", 2024-04-26T01:08:25.863 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.863 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.863 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.863 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.863 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.067197+0000", 2024-04-26T01:08:25.863 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1699999999999997e-07 2024-04-26T01:08:25.863 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.863 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.863 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.863 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.067201+0000", 2024-04-26T01:08:25.863 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5340000000000001e-06 2024-04-26T01:08:25.864 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.864 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.864 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.864 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.194638+0000", 2024-04-26T01:08:25.864 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.12743633600000001 2024-04-26T01:08:25.864 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.864 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.864 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.864 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.194653+0000", 2024-04-26T01:08:25.865 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5194000000000001e-05 2024-04-26T01:08:25.865 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.865 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.865 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.865 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.194810+0000", 2024-04-26T01:08:25.865 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015676500000000001 2024-04-26T01:08:25.865 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.865 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.865 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.865 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.194929+0000", 2024-04-26T01:08:25.865 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000119356 2024-04-26T01:08:25.865 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.865 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.865 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.866 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.866 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.866 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:549 3.bs0 3:d071ab05:::smithi16428812-44:head [append 0~442368] snapc e=[e,d,b,7] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-04-26T01:08:25.866 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:21.081171+0000", 2024-04-26T01:08:25.866 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.720186934, 2024-04-26T01:08:25.866 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.032989434999999998, 2024-04-26T01:08:25.866 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.866 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.866 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:25.866 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.866 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.866 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.866 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 549 2024-04-26T01:08:25.867 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.867 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.867 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.867 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.867 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.081171+0000", 2024-04-26T01:08:25.867 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.867 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.867 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.867 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.867 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.081171+0000", 2024-04-26T01:08:25.867 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.867 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.867 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.867 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.868 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.081174+0000", 2024-04-26T01:08:25.868 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6000000000000001e-06 2024-04-26T01:08:25.868 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.868 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.868 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.868 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.082287+0000", 2024-04-26T01:08:25.868 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0011128959999999999 2024-04-26T01:08:25.868 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.868 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.868 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.868 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.082313+0000", 2024-04-26T01:08:25.868 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5803999999999999e-05 2024-04-26T01:08:25.868 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.869 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.869 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.869 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.082329+0000", 2024-04-26T01:08:25.869 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6013000000000001e-05 2024-04-26T01:08:25.869 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.869 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.869 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.869 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.084819+0000", 2024-04-26T01:08:25.869 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024897600000000002 2024-04-26T01:08:25.869 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.869 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.869 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.869 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.085018+0000", 2024-04-26T01:08:25.869 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000199335 2024-04-26T01:08:25.869 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.870 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.870 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.870 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.086163+0000", 2024-04-26T01:08:25.870 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001144663 2024-04-26T01:08:25.870 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.870 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.870 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.870 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.106838+0000", 2024-04-26T01:08:25.870 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.020675487999999999 2024-04-26T01:08:25.870 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.870 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.870 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:25.870 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.114054+0000", 2024-04-26T01:08:25.871 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0072160560000000002 2024-04-26T01:08:25.871 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.871 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.871 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.871 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.114161+0000", 2024-04-26T01:08:25.871 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010682 2024-04-26T01:08:25.871 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.871 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.871 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.871 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.871 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.871 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:584 3.7s0 3:e6dd9561:::smithi16428812-19:head [append 0~688128] snapc f=[f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:25.871 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.157148+0000", 2024-04-26T01:08:25.871 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.644210387, 2024-04-26T01:08:25.871 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.038356399999999999, 2024-04-26T01:08:25.872 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.872 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.872 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:25.872 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.872 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.872 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.872 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 584 2024-04-26T01:08:25.872 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.872 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.872 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.872 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.872 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.157148+0000", 2024-04-26T01:08:25.872 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.873 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.873 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.873 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.873 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.157148+0000", 2024-04-26T01:08:25.873 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.873 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.873 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.873 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.873 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.157152+0000", 2024-04-26T01:08:25.873 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7670000000000001e-06 2024-04-26T01:08:25.873 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.873 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.873 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.873 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.159348+0000", 2024-04-26T01:08:25.874 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002196188 2024-04-26T01:08:25.874 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.874 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.874 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.874 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.159381+0000", 2024-04-26T01:08:25.874 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3479999999999998e-05 2024-04-26T01:08:25.874 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.874 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.874 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.874 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.159402+0000", 2024-04-26T01:08:25.874 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0945e-05 2024-04-26T01:08:25.874 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.874 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.874 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.875 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.159510+0000", 2024-04-26T01:08:25.875 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000107137 2024-04-26T01:08:25.875 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.875 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.875 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op must wait for map", 2024-04-26T01:08:25.875 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.159518+0000", 2024-04-26T01:08:25.875 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6149999999999997e-06 2024-04-26T01:08:25.875 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.875 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.875 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.875 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164312+0000", 2024-04-26T01:08:25.875 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0047940639999999998 2024-04-26T01:08:25.875 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.875 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.875 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.876 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164563+0000", 2024-04-26T01:08:25.876 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00025093199999999999 2024-04-26T01:08:25.876 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.876 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.876 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.876 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.168444+0000", 2024-04-26T01:08:25.876 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0038806689999999998 2024-04-26T01:08:25.876 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.876 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.876 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.876 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.183111+0000", 2024-04-26T01:08:25.876 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014667237 2024-04-26T01:08:25.876 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.876 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.877 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:25.877 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.195305+0000", 2024-04-26T01:08:25.877 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012194174 2024-04-26T01:08:25.877 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.877 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.877 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.877 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.195504+0000", 2024-04-26T01:08:25.877 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000199192 2024-04-26T01:08:25.877 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.877 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.877 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.877 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.877 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.877 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:585 3.7s0 3:e6dd9561:::smithi16428812-19:head [append 0~507904] snapc f=[f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:25.877 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.164838+0000", 2024-04-26T01:08:25.878 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.6365199239999999, 2024-04-26T01:08:25.878 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.031703666999999998, 2024-04-26T01:08:25.878 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.878 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.878 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:25.878 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.878 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.878 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.878 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 585 2024-04-26T01:08:25.878 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.878 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.878 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.878 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.879 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164838+0000", 2024-04-26T01:08:25.879 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.879 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.879 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.879 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.879 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164838+0000", 2024-04-26T01:08:25.879 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.879 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.879 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.879 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.879 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164841+0000", 2024-04-26T01:08:25.879 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9229999999999998e-06 2024-04-26T01:08:25.879 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.879 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.880 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.880 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.165724+0000", 2024-04-26T01:08:25.880 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00088260300000000003 2024-04-26T01:08:25.880 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.880 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.880 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.880 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.165754+0000", 2024-04-26T01:08:25.880 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9771999999999999e-05 2024-04-26T01:08:25.880 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.880 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.880 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.880 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.165774+0000", 2024-04-26T01:08:25.880 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0058999999999999e-05 2024-04-26T01:08:25.881 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.881 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.881 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.881 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.184152+0000", 2024-04-26T01:08:25.881 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018378627000000002 2024-04-26T01:08:25.881 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.881 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.881 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.881 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.184931+0000", 2024-04-26T01:08:25.881 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00077872800000000002 2024-04-26T01:08:25.881 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.881 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.881 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.881 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.187204+0000", 2024-04-26T01:08:25.882 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022730279999999999 2024-04-26T01:08:25.882 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.882 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.882 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.882 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.194601+0000", 2024-04-26T01:08:25.882 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0073969459999999997 2024-04-26T01:08:25.882 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.882 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.882 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:25.882 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.196160+0000", 2024-04-26T01:08:25.882 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015588819999999999 2024-04-26T01:08:25.882 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.882 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.882 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.882 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.196542+0000", 2024-04-26T01:08:25.883 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000382099 2024-04-26T01:08:25.883 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.883 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.883 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.883 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.883 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.883 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:586 3.7s0 3:e6dd9561:::smithi16428812-19:head [append 0~655360] snapc f=[f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:25.883 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.165808+0000", 2024-04-26T01:08:25.883 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.6355504239999998, 2024-04-26T01:08:25.883 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.031631711999999999, 2024-04-26T01:08:25.883 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.883 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.883 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:25.883 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.884 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.884 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.884 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 586 2024-04-26T01:08:25.884 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.884 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.884 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.884 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.884 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.165808+0000", 2024-04-26T01:08:25.884 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.884 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.884 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.884 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.884 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.165808+0000", 2024-04-26T01:08:25.884 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.885 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.885 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.885 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.885 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.165810+0000", 2024-04-26T01:08:25.885 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7859999999999999e-06 2024-04-26T01:08:25.885 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.885 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.885 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.885 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.174743+0000", 2024-04-26T01:08:25.885 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0089337259999999995 2024-04-26T01:08:25.885 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.885 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.885 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.885 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.174774+0000", 2024-04-26T01:08:25.886 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.046e-05 2024-04-26T01:08:25.886 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.886 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.886 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.886 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.174795+0000", 2024-04-26T01:08:25.886 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0579999999999999e-05 2024-04-26T01:08:25.886 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.886 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.886 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.886 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.188808+0000", 2024-04-26T01:08:25.886 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014013479000000001 2024-04-26T01:08:25.886 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.886 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.886 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.887 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.189488+0000", 2024-04-26T01:08:25.887 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00067966599999999999 2024-04-26T01:08:25.887 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.887 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.887 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.887 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.191510+0000", 2024-04-26T01:08:25.887 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0020227230000000001 2024-04-26T01:08:25.887 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.887 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.887 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.887 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.194823+0000", 2024-04-26T01:08:25.887 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033125580000000002 2024-04-26T01:08:25.887 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.887 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.888 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:25.888 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.197331+0000", 2024-04-26T01:08:25.888 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0025083240000000001 2024-04-26T01:08:25.888 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.888 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.888 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.888 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.197440+0000", 2024-04-26T01:08:25.888 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010841 2024-04-26T01:08:25.888 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.888 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.888 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.888 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.888 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.888 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:599 3.fs0 3:fa21e776:::smithi16428812-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:25.889 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.187146+0000", 2024-04-26T01:08:25.889 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.614212137, 2024-04-26T01:08:25.889 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.094712416999999993, 2024-04-26T01:08:25.889 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.889 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.889 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.889 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.889 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.889 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.889 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 599 2024-04-26T01:08:25.889 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.889 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.889 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.889 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.890 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.187146+0000", 2024-04-26T01:08:25.890 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.890 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.890 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.890 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.890 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.187146+0000", 2024-04-26T01:08:25.890 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.890 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.890 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.890 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.890 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.187147+0000", 2024-04-26T01:08:25.890 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7800000000000001e-07 2024-04-26T01:08:25.890 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.891 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.891 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.891 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.187151+0000", 2024-04-26T01:08:25.891 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.546e-06 2024-04-26T01:08:25.891 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.891 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.891 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.891 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.187161+0000", 2024-04-26T01:08:25.891 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0304e-05 2024-04-26T01:08:25.891 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.891 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.891 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.891 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.187166+0000", 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2429999999999996e-06 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.205675+0000", 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018509105000000001 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.205807+0000", 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013163899999999999 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.212468+0000", 2024-04-26T01:08:25.892 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0066608329999999997 2024-04-26T01:08:25.893 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.893 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.893 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.893 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.212545+0000", 2024-04-26T01:08:25.893 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7768999999999994e-05 2024-04-26T01:08:25.893 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.893 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.893 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.893 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.281859+0000", 2024-04-26T01:08:25.893 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.069313200000000005 2024-04-26T01:08:25.893 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.893 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.893 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.893 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.893 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 58/44 ECSubWriteReply(tid=310, last_complete=58'73, committed=1, applied=1))", 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.213740+0000", 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.587617992, 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.066820729999999995, 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.213740+0000", 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.894 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.213740+0000", 2024-04-26T01:08:25.895 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.895 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.895 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.895 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.895 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.213741+0000", 2024-04-26T01:08:25.895 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9299999999999999e-07 2024-04-26T01:08:25.895 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.895 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.895 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.895 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.213742+0000", 2024-04-26T01:08:25.895 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6130000000000001e-06 2024-04-26T01:08:25.895 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.895 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.895 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.896 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.279951+0000", 2024-04-26T01:08:25.896 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.066209009999999999 2024-04-26T01:08:25.896 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.896 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.896 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.896 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.279960+0000", 2024-04-26T01:08:25.896 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8440000000000004e-06 2024-04-26T01:08:25.896 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.896 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.896 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.896 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.280027+0000", 2024-04-26T01:08:25.896 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6778000000000003e-05 2024-04-26T01:08:25.896 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.896 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.896 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.897 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.280561+0000", 2024-04-26T01:08:25.897 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00053429200000000001 2024-04-26T01:08:25.897 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.897 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.897 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.897 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.897 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.897 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:601 3.bs0 3:d5691d7a:::smithi16428812-40:head [copy-from ver 4] snapc f=[f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:25.897 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.340640+0000", 2024-04-26T01:08:25.897 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.4607184090000001, 2024-04-26T01:08:25.897 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.78325740899999996, 2024-04-26T01:08:25.897 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.897 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.897 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:25.898 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.898 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.898 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.898 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 601 2024-04-26T01:08:25.898 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.898 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.898 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.898 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.898 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340640+0000", 2024-04-26T01:08:25.898 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.898 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.898 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.898 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.898 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340640+0000", 2024-04-26T01:08:25.899 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.899 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.899 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.899 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.899 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340644+0000", 2024-04-26T01:08:25.899 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9169999999999999e-06 2024-04-26T01:08:25.899 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.899 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.899 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.899 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340661+0000", 2024-04-26T01:08:25.899 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7390000000000001e-05 2024-04-26T01:08:25.899 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.899 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.899 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.899 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340696+0000", 2024-04-26T01:08:25.900 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4332999999999998e-05 2024-04-26T01:08:25.900 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.900 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.900 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.900 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340728+0000", 2024-04-26T01:08:25.900 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.2246000000000003e-05 2024-04-26T01:08:25.900 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.900 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.900 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.900 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340944+0000", 2024-04-26T01:08:25.900 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00021597700000000001 2024-04-26T01:08:25.900 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.900 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.900 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.901 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341149+0000", 2024-04-26T01:08:25.901 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000204725 2024-04-26T01:08:25.901 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.901 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.901 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:25.901 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.360012+0000", 2024-04-26T01:08:25.901 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018863683999999999 2024-04-26T01:08:25.901 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.901 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.901 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:25.901 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.362478+0000", 2024-04-26T01:08:25.901 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024654849999999999 2024-04-26T01:08:25.901 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.902 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.902 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:25.902 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.123741+0000", 2024-04-26T01:08:25.902 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.76126282099999998 2024-04-26T01:08:25.902 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.902 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.902 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.902 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.123897+0000", 2024-04-26T01:08:25.902 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015683099999999999 2024-04-26T01:08:25.902 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.902 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.902 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.902 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.903 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.903 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:610 3.fs0 3:fe09fb9c:::smithi16428812-49:head [read 0~4276224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:25.903 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.341162+0000", 2024-04-26T01:08:25.903 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.4601966260000001, 2024-04-26T01:08:25.903 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.030509089, 2024-04-26T01:08:25.903 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:25.903 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:25.903 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:25.903 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:25.903 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:25.903 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:25.903 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 610 2024-04-26T01:08:25.904 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.904 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:25.904 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.904 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:25.904 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341162+0000", 2024-04-26T01:08:25.904 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.904 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.904 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.904 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:25.904 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341162+0000", 2024-04-26T01:08:25.904 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:25.904 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.904 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.905 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:25.905 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341163+0000", 2024-04-26T01:08:25.905 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.17e-06 2024-04-26T01:08:25.905 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.905 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.905 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:25.905 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341170+0000", 2024-04-26T01:08:25.905 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2799999999999998e-06 2024-04-26T01:08:25.905 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.905 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.905 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:25.905 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341186+0000", 2024-04-26T01:08:25.905 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6252e-05 2024-04-26T01:08:25.906 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.906 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.906 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:25.906 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341197+0000", 2024-04-26T01:08:25.906 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0327e-05 2024-04-26T01:08:25.906 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.906 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.906 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:25.906 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.344020+0000", 2024-04-26T01:08:25.906 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0028235259999999998 2024-04-26T01:08:25.906 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.906 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.906 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:25.906 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.344131+0000", 2024-04-26T01:08:25.907 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000110278 2024-04-26T01:08:25.907 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:25.907 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:25.907 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:25.907 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.371671+0000", 2024-04-26T01:08:25.907 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.027540255999999999 2024-04-26T01:08:25.907 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.907 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.907 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.907 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:25.907 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:25.907 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:25.909 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:08:25.909 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:25.904+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:26.009 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:26.004+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:26.081 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:26.081 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:26.081 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-04-26T01:08:26.081 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:26.095 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:08:26.138 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:26.136+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:26.167 INFO:tasks.rados.rados.0.smithi164.stdout:239: done (7 left) 2024-04-26T01:08:26.168 INFO:tasks.rados.rados.0.smithi164.stdout:246: done (6 left) 2024-04-26T01:08:26.168 INFO:tasks.rados.rados.0.smithi164.stdout:247: read oid 7 snap -1 2024-04-26T01:08:26.168 INFO:tasks.rados.rados.0.smithi164.stdout:247: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-26T01:08:26.168 INFO:tasks.rados.rados.0.smithi164.stdout:248: delete oid 38 current snap is 14 2024-04-26T01:08:26.168 INFO:tasks.rados.rados.0.smithi164.stdout:240: finishing rollback tid 2 to smithi16428812-29 2024-04-26T01:08:26.168 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 29 v 40 (ObjNum 98 snap 11 seq_num 98) dirty exists 2024-04-26T01:08:26.168 INFO:tasks.rados.rados.0.smithi164.stdout:241: finishing write tid 1 to smithi16428812-44 2024-04-26T01:08:26.168 INFO:tasks.rados.rados.0.smithi164.stdout:241: finishing write tid 2 to smithi16428812-44 2024-04-26T01:08:26.168 INFO:tasks.rados.rados.0.smithi164.stdout:241: finishing write tid 3 to smithi16428812-44 2024-04-26T01:08:26.168 INFO:tasks.rados.rados.0.smithi164.stdout:241: finishing write tid 4 to smithi16428812-44 2024-04-26T01:08:26.168 INFO:tasks.rados.rados.0.smithi164.stdout:241: finishing write tid 5 to smithi16428812-44 2024-04-26T01:08:26.168 INFO:tasks.rados.rados.0.smithi164.stdout:241: oid 44 updating version 0 to 49 2024-04-26T01:08:26.169 INFO:tasks.rados.rados.0.smithi164.stdout:241: oid 44 version 49 is already newer than 48 2024-04-26T01:08:26.169 INFO:tasks.rados.rados.0.smithi164.stdout:241: oid 44 updating version 49 to 50 2024-04-26T01:08:26.169 INFO:tasks.rados.rados.0.smithi164.stdout:241: oid 44 version 50 is already newer than 47 2024-04-26T01:08:26.169 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 44 v 50 (ObjNum 113 snap 14 seq_num 113) dirty exists 2024-04-26T01:08:26.169 INFO:tasks.rados.rados.0.smithi164.stdout:241: left oid 44 (ObjNum 113 snap 14 seq_num 113) 2024-04-26T01:08:26.169 INFO:tasks.rados.rados.0.smithi164.stdout:242: finishing write tid 1 to smithi16428812-25 2024-04-26T01:08:26.169 INFO:tasks.rados.rados.0.smithi164.stdout:242: finishing write tid 2 to smithi16428812-25 2024-04-26T01:08:26.169 INFO:tasks.rados.rados.0.smithi164.stdout:243: expect (ObjNum 91 snap 7 seq_num 91) 2024-04-26T01:08:26.181 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-26T01:08:26.211 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:26.207+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:26.260 INFO:tasks.rados.rados.0.smithi164.stdout:242: finishing write tid 3 to smithi16428812-25 2024-04-26T01:08:26.260 INFO:tasks.rados.rados.0.smithi164.stdout:242: finishing write tid 4 to smithi16428812-25 2024-04-26T01:08:26.260 INFO:tasks.rados.rados.0.smithi164.stdout:242: finishing write tid 5 to smithi16428812-25 2024-04-26T01:08:26.260 INFO:tasks.rados.rados.0.smithi164.stdout:242: oid 25 updating version 0 to 70 2024-04-26T01:08:26.260 INFO:tasks.rados.rados.0.smithi164.stdout:242: oid 25 updating version 70 to 72 2024-04-26T01:08:26.260 INFO:tasks.rados.rados.0.smithi164.stdout:242: oid 25 version 72 is already newer than 71 2024-04-26T01:08:26.260 INFO:tasks.rados.rados.0.smithi164.stdout:242: oid 25 version 72 is already newer than 69 2024-04-26T01:08:26.260 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 25 v 72 (ObjNum 114 snap 14 seq_num 114) dirty exists 2024-04-26T01:08:26.261 INFO:tasks.rados.rados.0.smithi164.stdout:242: left oid 25 (ObjNum 114 snap 14 seq_num 114) 2024-04-26T01:08:26.261 INFO:tasks.rados.rados.0.smithi164.stdout:244: expect (ObjNum 99 snap 11 seq_num 99) 2024-04-26T01:08:26.285 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:26.285 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:26.285 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-04-26T01:08:26.285 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-04-26T01:08:26.285 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:26.300 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:08:26.305 INFO:tasks.rados.rados.0.smithi164.stdout:245: finishing write tid 1 to smithi16428812-23 2024-04-26T01:08:26.305 INFO:tasks.rados.rados.0.smithi164.stdout:245: finishing write tid 2 to smithi16428812-23 2024-04-26T01:08:26.305 INFO:tasks.rados.rados.0.smithi164.stdout:245: finishing write tid 3 to smithi16428812-23 2024-04-26T01:08:26.305 INFO:tasks.rados.rados.0.smithi164.stdout:245: finishing write tid 4 to smithi16428812-23 2024-04-26T01:08:26.305 INFO:tasks.rados.rados.0.smithi164.stdout:245: finishing write tid 5 to smithi16428812-23 2024-04-26T01:08:26.305 INFO:tasks.rados.rados.0.smithi164.stdout:245: oid 23 updating version 0 to 56 2024-04-26T01:08:26.305 INFO:tasks.rados.rados.0.smithi164.stdout:245: oid 23 version 56 is already newer than 54 2024-04-26T01:08:26.305 INFO:tasks.rados.rados.0.smithi164.stdout:245: oid 23 version 56 is already newer than 55 2024-04-26T01:08:26.305 INFO:tasks.rados.rados.0.smithi164.stdout:245: oid 23 version 56 is already newer than 53 2024-04-26T01:08:26.305 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 23 v 56 (ObjNum 115 snap 14 seq_num 115) dirty exists 2024-04-26T01:08:26.306 INFO:tasks.rados.rados.0.smithi164.stdout:245: left oid 23 (ObjNum 115 snap 14 seq_num 115) 2024-04-26T01:08:26.306 INFO:tasks.rados.rados.0.smithi164.stdout:247: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-26T01:08:26.313 INFO:tasks.rados.rados.0.smithi164.stdout:240: done (7 left) 2024-04-26T01:08:26.313 INFO:tasks.rados.rados.0.smithi164.stdout:241: done (6 left) 2024-04-26T01:08:26.313 INFO:tasks.rados.rados.0.smithi164.stdout:242: done (5 left) 2024-04-26T01:08:26.313 INFO:tasks.rados.rados.0.smithi164.stdout:243: done (4 left) 2024-04-26T01:08:26.313 INFO:tasks.rados.rados.0.smithi164.stdout:244: done (3 left) 2024-04-26T01:08:26.313 INFO:tasks.rados.rados.0.smithi164.stdout:245: done (2 left) 2024-04-26T01:08:26.313 INFO:tasks.rados.rados.0.smithi164.stdout:247: done (1 left) 2024-04-26T01:08:26.313 INFO:tasks.rados.rados.0.smithi164.stdout:248: done (0 left) 2024-04-26T01:08:26.313 INFO:tasks.rados.rados.0.smithi164.stdout:249: snap_remove snap 13 2024-04-26T01:08:26.350 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:26.348+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:26.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:26.412+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:26.422 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:26.422 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:26.422 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:26.422 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:26.422 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.422 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:16 3.4s0 3:2026e263:::smithi16428812-6:head [append 0~98304,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:26.422 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.307585+0000", 2024-04-26T01:08:26.422 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 27.111840036, 2024-04-26T01:08:26.423 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.016360486, 2024-04-26T01:08:26.423 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.423 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.423 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:26.423 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.423 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.423 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.423 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 16 2024-04-26T01:08:26.423 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.423 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.423 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.423 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.423 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307585+0000", 2024-04-26T01:08:26.424 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.424 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.424 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.424 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.424 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307585+0000", 2024-04-26T01:08:26.424 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.424 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.424 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.424 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.424 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307587+0000", 2024-04-26T01:08:26.424 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1280000000000002e-06 2024-04-26T01:08:26.424 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.424 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.424 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.425 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308008+0000", 2024-04-26T01:08:26.425 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00042080399999999999 2024-04-26T01:08:26.425 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.425 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.425 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.425 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308031+0000", 2024-04-26T01:08:26.425 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3085999999999999e-05 2024-04-26T01:08:26.425 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.425 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.425 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.425 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308041+0000", 2024-04-26T01:08:26.425 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9650000000000003e-06 2024-04-26T01:08:26.425 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.425 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.426 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.426 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308110+0000", 2024-04-26T01:08:26.426 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.915e-05 2024-04-26T01:08:26.426 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.426 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.426 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:26.426 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308277+0000", 2024-04-26T01:08:26.426 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016695800000000001 2024-04-26T01:08:26.426 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.426 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.426 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:26.426 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.309191+0000", 2024-04-26T01:08:26.426 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00091407599999999999 2024-04-26T01:08:26.426 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.427 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.427 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:26.427 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314046+0000", 2024-04-26T01:08:26.427 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.004854377 2024-04-26T01:08:26.427 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.427 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.427 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:26.427 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323807+0000", 2024-04-26T01:08:26.427 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0097610869999999999 2024-04-26T01:08:26.427 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.427 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.428 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.428 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323946+0000", 2024-04-26T01:08:26.428 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013885500000000001 2024-04-26T01:08:26.428 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.428 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.428 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.428 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.428 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.428 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:17 3.4s0 3:2026e263:::smithi16428812-6:head [setxattr _header (55) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:26.428 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.335986+0000", 2024-04-26T01:08:26.428 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 27.083438829999999, 2024-04-26T01:08:26.428 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010264923, 2024-04-26T01:08:26.428 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.428 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.429 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:26.429 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.429 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.429 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.429 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 17 2024-04-26T01:08:26.429 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.429 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.429 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.429 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.429 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.335986+0000", 2024-04-26T01:08:26.429 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.429 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.429 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.429 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.430 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.335986+0000", 2024-04-26T01:08:26.430 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.430 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.430 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.430 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.430 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.335988+0000", 2024-04-26T01:08:26.430 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.373e-06 2024-04-26T01:08:26.430 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.430 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.430 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.430 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.335994+0000", 2024-04-26T01:08:26.430 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0700000000000003e-06 2024-04-26T01:08:26.430 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.431 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.431 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.431 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336007+0000", 2024-04-26T01:08:26.431 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3577000000000001e-05 2024-04-26T01:08:26.431 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.431 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.431 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.431 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336018+0000", 2024-04-26T01:08:26.431 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0668e-05 2024-04-26T01:08:26.431 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.432 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.432 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.432 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336285+0000", 2024-04-26T01:08:26.432 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026682799999999999 2024-04-26T01:08:26.432 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.432 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.432 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:26.432 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336439+0000", 2024-04-26T01:08:26.432 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000154048 2024-04-26T01:08:26.432 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.432 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.432 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:26.432 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.338115+0000", 2024-04-26T01:08:26.432 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0016757880000000001 2024-04-26T01:08:26.433 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.433 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.433 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:26.433 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.339187+0000", 2024-04-26T01:08:26.433 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001072494 2024-04-26T01:08:26.433 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.433 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.433 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:26.433 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.346146+0000", 2024-04-26T01:08:26.433 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0069590279999999999 2024-04-26T01:08:26.433 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.433 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.433 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.433 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.346251+0000", 2024-04-26T01:08:26.433 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000105049 2024-04-26T01:08:26.434 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.434 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.434 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.434 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.434 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.434 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:18 3.4s0 3:2026e263:::smithi16428812-6:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:26.434 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.336046+0000", 2024-04-26T01:08:26.434 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 27.083378906, 2024-04-26T01:08:26.434 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013179965, 2024-04-26T01:08:26.434 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.434 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.434 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:26.434 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.434 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.435 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.435 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 18 2024-04-26T01:08:26.435 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.435 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.435 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.435 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.435 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336046+0000", 2024-04-26T01:08:26.435 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.435 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.435 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.435 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.435 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336046+0000", 2024-04-26T01:08:26.436 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.436 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.436 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.436 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.436 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336047+0000", 2024-04-26T01:08:26.436 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1119999999999999e-06 2024-04-26T01:08:26.436 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.436 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.436 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.436 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336052+0000", 2024-04-26T01:08:26.436 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3510000000000004e-06 2024-04-26T01:08:26.436 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.437 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.437 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.437 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336063+0000", 2024-04-26T01:08:26.437 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1722999999999999e-05 2024-04-26T01:08:26.437 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.437 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.437 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.437 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336070+0000", 2024-04-26T01:08:26.437 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6259999999999997e-06 2024-04-26T01:08:26.437 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.437 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.437 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.437 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.338893+0000", 2024-04-26T01:08:26.438 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0028230009999999999 2024-04-26T01:08:26.438 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.438 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.438 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:26.438 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.339060+0000", 2024-04-26T01:08:26.438 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016734700000000001 2024-04-26T01:08:26.438 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.438 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.438 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.438 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.346619+0000", 2024-04-26T01:08:26.438 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0075586949999999998 2024-04-26T01:08:26.438 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.438 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.438 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:26.438 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.346731+0000", 2024-04-26T01:08:26.439 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000111863 2024-04-26T01:08:26.439 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.439 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.439 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.439 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349226+0000", 2024-04-26T01:08:26.439 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024952469999999999 2024-04-26T01:08:26.439 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.439 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.439 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.439 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.439 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.439 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.5s1 42/30 ECSubWrite(tid=32, reqid=client.4521.0:351, at_version=42'28, trim_to=0'0, roll_forward_to=42'25))", 2024-04-26T01:08:26.439 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:08.417770+0000", 2024-04-26T01:08:26.440 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 18.001655037999999, 2024-04-26T01:08:26.440 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.64597556300000003, 2024-04-26T01:08:26.440 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.440 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.440 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:26.440 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.440 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.440 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.440 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.417770+0000", 2024-04-26T01:08:26.440 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.440 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.440 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.440 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.440 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.417770+0000", 2024-04-26T01:08:26.441 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.441 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.441 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.441 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.441 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.417771+0000", 2024-04-26T01:08:26.441 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6499999999999998e-07 2024-04-26T01:08:26.441 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.441 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.441 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.441 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.417789+0000", 2024-04-26T01:08:26.441 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8198000000000001e-05 2024-04-26T01:08:26.441 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.441 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.441 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.442 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.062086+0000", 2024-04-26T01:08:26.442 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.64429725800000004 2024-04-26T01:08:26.442 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.442 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.442 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.442 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.062104+0000", 2024-04-26T01:08:26.442 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8053000000000001e-05 2024-04-26T01:08:26.442 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.442 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.442 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.442 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.062287+0000", 2024-04-26T01:08:26.442 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00018218499999999999 2024-04-26T01:08:26.442 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.442 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.443 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:26.443 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.062316+0000", 2024-04-26T01:08:26.443 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9547e-05 2024-04-26T01:08:26.443 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.443 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.443 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:26.443 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.063675+0000", 2024-04-26T01:08:26.443 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001358831 2024-04-26T01:08:26.443 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.443 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.443 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.443 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.063746+0000", 2024-04-26T01:08:26.443 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0625999999999998e-05 2024-04-26T01:08:26.443 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.444 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.444 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.444 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.444 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.444 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(osd.14.21:2 3.cs1 3.83402d7c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:26.444 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.432903+0000", 2024-04-26T01:08:26.444 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 16.986522477000001, 2024-04-26T01:08:26.444 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97310677899999998, 2024-04-26T01:08:26.444 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.444 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.444 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:26.444 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.444 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.445 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.445 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432903+0000", 2024-04-26T01:08:26.445 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.445 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.445 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.445 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.445 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432903+0000", 2024-04-26T01:08:26.445 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.445 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.445 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.445 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.445 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432904+0000", 2024-04-26T01:08:26.445 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4839999999999999e-06 2024-04-26T01:08:26.445 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.446 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.446 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.446 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432912+0000", 2024-04-26T01:08:26.446 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3489999999999992e-06 2024-04-26T01:08:26.446 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.446 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.446 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.446 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432927+0000", 2024-04-26T01:08:26.446 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4297000000000001e-05 2024-04-26T01:08:26.446 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.446 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.446 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.446 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432935+0000", 2024-04-26T01:08:26.447 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0360000000000006e-06 2024-04-26T01:08:26.447 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.447 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.447 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.447 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432997+0000", 2024-04-26T01:08:26.447 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2176999999999998e-05 2024-04-26T01:08:26.447 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.447 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.447 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:26.447 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.433000+0000", 2024-04-26T01:08:26.447 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1990000000000002e-06 2024-04-26T01:08:26.447 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.447 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.447 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.448 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.405993+0000", 2024-04-26T01:08:26.448 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97299321100000002 2024-04-26T01:08:26.448 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.448 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.448 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.448 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406009+0000", 2024-04-26T01:08:26.448 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6025999999999999e-05 2024-04-26T01:08:26.448 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.448 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.448 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.448 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.448 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.448 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:375 3.cs1 3.33e19ddc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:26.448 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.436798+0000", 2024-04-26T01:08:26.448 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 16.982627082, 2024-04-26T01:08:26.449 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96902882700000004, 2024-04-26T01:08:26.449 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.449 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.449 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:26.449 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.449 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.449 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.449 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 375 2024-04-26T01:08:26.449 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.449 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.449 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.449 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.449 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436798+0000", 2024-04-26T01:08:26.449 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.449 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.450 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.450 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.450 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436798+0000", 2024-04-26T01:08:26.450 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.450 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.450 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.450 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.450 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436800+0000", 2024-04-26T01:08:26.450 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.756e-06 2024-04-26T01:08:26.451 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.451 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.451 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.451 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436810+0000", 2024-04-26T01:08:26.451 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9860000000000006e-06 2024-04-26T01:08:26.451 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.451 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.451 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.451 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436834+0000", 2024-04-26T01:08:26.451 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4457e-05 2024-04-26T01:08:26.451 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.451 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.451 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.452 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436846+0000", 2024-04-26T01:08:26.452 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1963999999999999e-05 2024-04-26T01:08:26.452 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.452 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.452 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.452 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436916+0000", 2024-04-26T01:08:26.452 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9667999999999998e-05 2024-04-26T01:08:26.452 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.452 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.452 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:26.452 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436918+0000", 2024-04-26T01:08:26.452 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.294e-06 2024-04-26T01:08:26.452 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.452 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.452 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.453 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.405810+0000", 2024-04-26T01:08:26.453 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96889232599999997 2024-04-26T01:08:26.453 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.453 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.453 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.453 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.405827+0000", 2024-04-26T01:08:26.453 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6376e-05 2024-04-26T01:08:26.453 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.453 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.453 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.453 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.453 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.453 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:376 3.cs1 3.33e19ddc (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:26.453 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.436879+0000", 2024-04-26T01:08:26.454 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 16.982545744999999, 2024-04-26T01:08:26.454 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96918375199999995, 2024-04-26T01:08:26.454 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.454 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.454 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:26.454 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.454 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.454 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.454 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 376 2024-04-26T01:08:26.454 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.454 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.454 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.454 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.454 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436879+0000", 2024-04-26T01:08:26.455 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.455 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.455 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.455 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.455 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436879+0000", 2024-04-26T01:08:26.455 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.455 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.455 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.455 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.455 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436880+0000", 2024-04-26T01:08:26.455 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0100000000000004e-07 2024-04-26T01:08:26.455 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.455 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.455 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.455 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436885+0000", 2024-04-26T01:08:26.456 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3959999999999999e-06 2024-04-26T01:08:26.456 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.456 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.456 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.456 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436896+0000", 2024-04-26T01:08:26.456 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1291000000000001e-05 2024-04-26T01:08:26.456 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.456 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.456 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.456 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436904+0000", 2024-04-26T01:08:26.456 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6000000000000007e-06 2024-04-26T01:08:26.456 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.456 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.456 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.456 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436935+0000", 2024-04-26T01:08:26.457 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0626000000000001e-05 2024-04-26T01:08:26.457 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.457 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.457 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:26.457 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436936+0000", 2024-04-26T01:08:26.457 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.44e-07 2024-04-26T01:08:26.457 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.457 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.457 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.457 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406050+0000", 2024-04-26T01:08:26.457 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96911480699999997 2024-04-26T01:08:26.457 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.457 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.457 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.457 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406063+0000", 2024-04-26T01:08:26.458 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2687000000000001e-05 2024-04-26T01:08:26.458 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.458 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.458 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.458 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.458 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.458 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:380 3.cs1 3.f9ba2d2c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:26.458 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.444825+0000", 2024-04-26T01:08:26.458 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 16.974600082999999, 2024-04-26T01:08:26.458 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96128573799999995, 2024-04-26T01:08:26.458 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.458 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.458 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:26.459 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.459 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.459 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.459 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 380 2024-04-26T01:08:26.459 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.459 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.459 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.459 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.459 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.444825+0000", 2024-04-26T01:08:26.459 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.459 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.459 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.459 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.459 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.444825+0000", 2024-04-26T01:08:26.460 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.460 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.460 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.460 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.460 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.444827+0000", 2024-04-26T01:08:26.460 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.731e-06 2024-04-26T01:08:26.460 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.460 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.460 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.460 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.445681+0000", 2024-04-26T01:08:26.460 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00085380000000000005 2024-04-26T01:08:26.460 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.460 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.461 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.461 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.445706+0000", 2024-04-26T01:08:26.461 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5768000000000001e-05 2024-04-26T01:08:26.461 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.461 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.461 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.461 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.445720+0000", 2024-04-26T01:08:26.461 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3788000000000001e-05 2024-04-26T01:08:26.461 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.461 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.462 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.462 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.445801+0000", 2024-04-26T01:08:26.462 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1048000000000001e-05 2024-04-26T01:08:26.462 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.462 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.462 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:26.463 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.445805+0000", 2024-04-26T01:08:26.463 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7479999999999999e-06 2024-04-26T01:08:26.463 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.463 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.463 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.463 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406098+0000", 2024-04-26T01:08:26.464 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96029355199999999 2024-04-26T01:08:26.464 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.464 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.464 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.464 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406111+0000", 2024-04-26T01:08:26.464 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2303000000000001e-05 2024-04-26T01:08:26.464 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.464 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.464 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.464 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.464 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.464 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:381 3.cs1 3.f9ba2d2c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:26.465 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.450315+0000", 2024-04-26T01:08:26.465 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 16.969109766999999, 2024-04-26T01:08:26.465 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95584751300000004, 2024-04-26T01:08:26.465 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.465 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.465 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:26.465 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.465 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.465 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.465 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 381 2024-04-26T01:08:26.465 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.465 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.465 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.466 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.466 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.450315+0000", 2024-04-26T01:08:26.466 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.466 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.466 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.466 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.466 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.450315+0000", 2024-04-26T01:08:26.466 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.466 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.467 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.467 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.467 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.450317+0000", 2024-04-26T01:08:26.467 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.271e-06 2024-04-26T01:08:26.467 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.467 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.467 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.467 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451037+0000", 2024-04-26T01:08:26.467 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00071991199999999996 2024-04-26T01:08:26.467 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.467 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.467 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.467 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451053+0000", 2024-04-26T01:08:26.467 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6049000000000002e-05 2024-04-26T01:08:26.468 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.468 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.468 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.468 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451060+0000", 2024-04-26T01:08:26.468 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8839999999999994e-06 2024-04-26T01:08:26.468 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.468 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.468 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.468 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451144+0000", 2024-04-26T01:08:26.468 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3251999999999995e-05 2024-04-26T01:08:26.468 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.468 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.469 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:26.469 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451147+0000", 2024-04-26T01:08:26.469 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.994e-06 2024-04-26T01:08:26.469 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.469 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.469 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.469 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406150+0000", 2024-04-26T01:08:26.470 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95500295999999996 2024-04-26T01:08:26.470 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.470 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.470 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.470 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406163+0000", 2024-04-26T01:08:26.470 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3191e-05 2024-04-26T01:08:26.470 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.470 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.470 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.470 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.470 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.470 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:382 3.cs1 3.f9ba2d2c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:26.470 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.451079+0000", 2024-04-26T01:08:26.471 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 16.968346029999999, 2024-04-26T01:08:26.471 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95513123099999997, 2024-04-26T01:08:26.471 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.471 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.471 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:26.471 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.471 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.471 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.471 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 382 2024-04-26T01:08:26.471 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.471 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.471 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.471 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.471 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451079+0000", 2024-04-26T01:08:26.472 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.472 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.472 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.472 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.472 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451079+0000", 2024-04-26T01:08:26.472 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.472 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.472 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.472 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.472 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451080+0000", 2024-04-26T01:08:26.472 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9599999999999999e-07 2024-04-26T01:08:26.472 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.472 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.472 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.473 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451116+0000", 2024-04-26T01:08:26.473 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6797e-05 2024-04-26T01:08:26.473 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.473 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.473 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.473 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451137+0000", 2024-04-26T01:08:26.473 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0163e-05 2024-04-26T01:08:26.473 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.473 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.473 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.473 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451142+0000", 2024-04-26T01:08:26.473 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6450000000000002e-06 2024-04-26T01:08:26.473 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.473 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.473 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.474 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451200+0000", 2024-04-26T01:08:26.474 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7703e-05 2024-04-26T01:08:26.474 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.474 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.474 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:26.474 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451201+0000", 2024-04-26T01:08:26.474 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1200000000000001e-07 2024-04-26T01:08:26.474 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.474 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.474 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.474 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406198+0000", 2024-04-26T01:08:26.474 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95499736000000002 2024-04-26T01:08:26.474 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.475 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.475 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.475 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406210+0000", 2024-04-26T01:08:26.475 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2055000000000001e-05 2024-04-26T01:08:26.475 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.475 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.475 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.475 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.475 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.475 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:383 3.cs1 3.f9ba2d2c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:26.475 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.451164+0000", 2024-04-26T01:08:26.475 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 16.968260667999999, 2024-04-26T01:08:26.475 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95510495399999995, 2024-04-26T01:08:26.475 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.476 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.476 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:26.476 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.476 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.476 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.476 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 383 2024-04-26T01:08:26.476 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.476 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.476 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.476 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.477 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451164+0000", 2024-04-26T01:08:26.477 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.477 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.477 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.477 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.477 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451164+0000", 2024-04-26T01:08:26.477 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.477 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.477 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.477 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.477 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451165+0000", 2024-04-26T01:08:26.478 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.58e-07 2024-04-26T01:08:26.478 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.478 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.478 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.478 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451168+0000", 2024-04-26T01:08:26.478 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3000000000000002e-06 2024-04-26T01:08:26.478 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.478 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.478 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.478 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451174+0000", 2024-04-26T01:08:26.478 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8220000000000003e-06 2024-04-26T01:08:26.478 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.479 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.479 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.479 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451179+0000", 2024-04-26T01:08:26.479 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4700000000000004e-06 2024-04-26T01:08:26.479 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.479 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.479 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.479 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451233+0000", 2024-04-26T01:08:26.479 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4086999999999999e-05 2024-04-26T01:08:26.479 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.479 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.479 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:26.479 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451234+0000", 2024-04-26T01:08:26.479 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0580000000000001e-06 2024-04-26T01:08:26.479 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.480 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.480 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.480 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406253+0000", 2024-04-26T01:08:26.480 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95501902800000005 2024-04-26T01:08:26.480 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.480 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.480 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.480 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406269+0000", 2024-04-26T01:08:26.480 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6731e-05 2024-04-26T01:08:26.480 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.480 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.480 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.480 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.480 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.481 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:384 3.cs1 3.f9ba2d2c (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:26.481 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.451185+0000", 2024-04-26T01:08:26.481 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 16.968239994000001, 2024-04-26T01:08:26.481 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95528391300000004, 2024-04-26T01:08:26.481 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.481 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.481 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:26.481 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.481 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.481 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.481 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 384 2024-04-26T01:08:26.481 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.481 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.481 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.481 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.482 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451185+0000", 2024-04-26T01:08:26.482 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.482 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.482 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.482 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.482 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451185+0000", 2024-04-26T01:08:26.482 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.482 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.482 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.482 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.482 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451186+0000", 2024-04-26T01:08:26.482 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3599999999999999e-07 2024-04-26T01:08:26.483 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.483 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.483 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.483 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451187+0000", 2024-04-26T01:08:26.483 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.804e-06 2024-04-26T01:08:26.483 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.483 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.483 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.483 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451193+0000", 2024-04-26T01:08:26.483 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4060000000000004e-06 2024-04-26T01:08:26.483 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.483 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.484 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.484 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451197+0000", 2024-04-26T01:08:26.484 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.262e-06 2024-04-26T01:08:26.484 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.484 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.484 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.484 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451254+0000", 2024-04-26T01:08:26.484 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6635000000000002e-05 2024-04-26T01:08:26.484 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.484 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.484 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:26.484 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451254+0000", 2024-04-26T01:08:26.484 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6599999999999995e-07 2024-04-26T01:08:26.485 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.485 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.485 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.485 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406454+0000", 2024-04-26T01:08:26.485 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95519958699999996 2024-04-26T01:08:26.485 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.485 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.485 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.485 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406469+0000", 2024-04-26T01:08:26.486 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5017e-05 2024-04-26T01:08:26.486 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.486 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.486 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.486 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.486 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.486 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:483 3.4s0 3:2d34bde3:::smithi16428812-29:head [copy-from ver 0] snapc a=[a,8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-26T01:08:26.486 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:14.042718+0000", 2024-04-26T01:08:26.486 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.376707422999999, 2024-04-26T01:08:26.486 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015567493, 2024-04-26T01:08:26.486 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.487 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.487 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:26.487 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.487 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.487 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.487 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 483 2024-04-26T01:08:26.487 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.487 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.487 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.487 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.487 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042718+0000", 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042718+0000", 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042721+0000", 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.2100000000000002e-06 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042756+0000", 2024-04-26T01:08:26.488 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4972000000000003e-05 2024-04-26T01:08:26.489 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.489 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.489 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.489 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042793+0000", 2024-04-26T01:08:26.489 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7027000000000003e-05 2024-04-26T01:08:26.489 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.489 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.489 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.489 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042819+0000", 2024-04-26T01:08:26.489 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6359999999999998e-05 2024-04-26T01:08:26.489 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.489 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.489 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.489 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042991+0000", 2024-04-26T01:08:26.489 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017123400000000001 2024-04-26T01:08:26.490 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.490 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.490 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op must wait for map", 2024-04-26T01:08:26.490 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.043005+0000", 2024-04-26T01:08:26.490 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4414e-05 2024-04-26T01:08:26.490 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.490 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.490 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.490 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.047524+0000", 2024-04-26T01:08:26.490 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0045195349999999999 2024-04-26T01:08:26.490 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.490 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.491 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:26.491 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.047801+0000", 2024-04-26T01:08:26.491 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00027666799999999999 2024-04-26T01:08:26.491 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.491 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.491 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.491 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.058285+0000", 2024-04-26T01:08:26.491 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010484073 2024-04-26T01:08:26.491 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.491 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.491 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.491 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.491 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.492 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:484 3.4s0 3:2d34bde3:::smithi16428812-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-26T01:08:26.492 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:14.042874+0000", 2024-04-26T01:08:26.492 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.376551359, 2024-04-26T01:08:26.492 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015940147000000002, 2024-04-26T01:08:26.492 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.492 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.492 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:26.492 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.492 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.492 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.492 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 484 2024-04-26T01:08:26.492 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.492 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.493 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.493 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.493 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042874+0000", 2024-04-26T01:08:26.493 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.493 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.493 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.493 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.493 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042874+0000", 2024-04-26T01:08:26.493 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.493 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.493 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.493 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.494 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042875+0000", 2024-04-26T01:08:26.494 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.64e-06 2024-04-26T01:08:26.494 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.494 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.494 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.494 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042884+0000", 2024-04-26T01:08:26.494 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1100000000000003e-06 2024-04-26T01:08:26.494 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.494 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.494 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.494 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042909+0000", 2024-04-26T01:08:26.495 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5857e-05 2024-04-26T01:08:26.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.495 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.495 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.495 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042929+0000", 2024-04-26T01:08:26.495 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9167000000000001e-05 2024-04-26T01:08:26.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.495 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.495 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.495 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.043135+0000", 2024-04-26T01:08:26.495 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00020666799999999999 2024-04-26T01:08:26.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.495 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.495 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.043158+0000", 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.304e-05 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.048261+0000", 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0051031039999999998 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for blocked object", 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.048407+0000", 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000145308 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.496 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.058472+0000", 2024-04-26T01:08:26.497 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010065703 2024-04-26T01:08:26.497 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.497 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.497 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:26.497 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.058641+0000", 2024-04-26T01:08:26.497 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016832199999999999 2024-04-26T01:08:26.497 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.497 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.497 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.497 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.058814+0000", 2024-04-26T01:08:26.497 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017322799999999999 2024-04-26T01:08:26.498 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.498 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.498 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.498 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.498 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.498 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:605 3.4s0 3:2d34bde3:::smithi16428812-29:head [read 0~2129920,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:26.498 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.340722+0000", 2024-04-26T01:08:26.498 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.0787026800000001, 2024-04-26T01:08:26.498 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010360348, 2024-04-26T01:08:26.498 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.498 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.498 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:26.498 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.498 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.499 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.499 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 605 2024-04-26T01:08:26.499 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.499 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.499 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.499 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.499 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340722+0000", 2024-04-26T01:08:26.499 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.499 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.499 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.499 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.499 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340722+0000", 2024-04-26T01:08:26.499 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.499 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.499 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.500 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.500 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340726+0000", 2024-04-26T01:08:26.500 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.467e-06 2024-04-26T01:08:26.500 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.500 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.500 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.500 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340743+0000", 2024-04-26T01:08:26.500 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7567e-05 2024-04-26T01:08:26.500 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.500 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.500 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.500 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340789+0000", 2024-04-26T01:08:26.500 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5259999999999997e-05 2024-04-26T01:08:26.500 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.501 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.501 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.501 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340812+0000", 2024-04-26T01:08:26.501 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2861999999999999e-05 2024-04-26T01:08:26.501 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.501 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.501 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.501 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340971+0000", 2024-04-26T01:08:26.501 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000159252 2024-04-26T01:08:26.501 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.501 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.501 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:26.501 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341112+0000", 2024-04-26T01:08:26.502 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000141349 2024-04-26T01:08:26.502 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.502 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.502 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.502 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.351083+0000", 2024-04-26T01:08:26.502 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0099705909999999991 2024-04-26T01:08:26.502 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.502 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.502 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.502 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.502 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.502 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:606 3.4s0 3:2d34bde3:::smithi16428812-29:head [read 0~2129920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:26.502 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.340889+0000", 2024-04-26T01:08:26.502 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.0785358650000001, 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012634348, 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 606 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340889+0000", 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.503 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340889+0000", 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340890+0000", 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6299999999999993e-07 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340896+0000", 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6500000000000001e-06 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.504 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.505 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340917+0000", 2024-04-26T01:08:26.505 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0769999999999999e-05 2024-04-26T01:08:26.505 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.505 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.505 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.505 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340927+0000", 2024-04-26T01:08:26.505 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0077e-05 2024-04-26T01:08:26.505 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.505 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.505 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.505 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341566+0000", 2024-04-26T01:08:26.505 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000638881 2024-04-26T01:08:26.505 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.505 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.505 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:26.506 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341677+0000", 2024-04-26T01:08:26.506 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000111595 2024-04-26T01:08:26.506 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.506 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.506 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.506 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.353524+0000", 2024-04-26T01:08:26.506 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011846412000000001 2024-04-26T01:08:26.506 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.506 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.506 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.506 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.506 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.506 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:607 3.4s0 3:2d34bde3:::smithi16428812-29:head [read 0~2129920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:26.506 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.341454+0000", 2024-04-26T01:08:26.507 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.0779713800000001, 2024-04-26T01:08:26.507 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014935789, 2024-04-26T01:08:26.507 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.507 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.507 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:26.507 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.507 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.507 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.507 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 607 2024-04-26T01:08:26.507 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.507 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.507 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.507 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.507 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341454+0000", 2024-04-26T01:08:26.508 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.508 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.508 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.508 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.508 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341454+0000", 2024-04-26T01:08:26.508 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.508 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.508 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.508 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.508 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341455+0000", 2024-04-26T01:08:26.508 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0550000000000001e-06 2024-04-26T01:08:26.509 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.509 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.509 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.509 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341464+0000", 2024-04-26T01:08:26.509 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4350000000000003e-06 2024-04-26T01:08:26.509 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.509 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.509 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.509 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341477+0000", 2024-04-26T01:08:26.509 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2731999999999999e-05 2024-04-26T01:08:26.509 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.509 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.509 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.509 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341488+0000", 2024-04-26T01:08:26.509 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.118e-05 2024-04-26T01:08:26.510 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.510 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.510 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.510 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.343260+0000", 2024-04-26T01:08:26.510 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0017719980000000001 2024-04-26T01:08:26.510 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.510 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.510 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:26.510 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.343498+0000", 2024-04-26T01:08:26.510 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00023756600000000001 2024-04-26T01:08:26.510 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.510 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.511 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.511 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.356390+0000", 2024-04-26T01:08:26.511 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012891823 2024-04-26T01:08:26.511 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.511 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.511 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.511 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.511 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.511 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:614 3.4s0 3:2d34bde3:::smithi16428812-29:head [rollback f] snapc f=[f,d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-26T01:08:26.511 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:25.399082+0000", 2024-04-26T01:08:26.511 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.02034336, 2024-04-26T01:08:26.511 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010883195, 2024-04-26T01:08:26.511 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.511 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.512 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:26.512 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.512 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.512 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.512 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 614 2024-04-26T01:08:26.512 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.512 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.512 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.512 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.512 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399082+0000", 2024-04-26T01:08:26.512 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.512 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.512 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.512 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.512 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399082+0000", 2024-04-26T01:08:26.513 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.513 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.513 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.513 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.513 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399082+0000", 2024-04-26T01:08:26.513 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5000000000000003e-07 2024-04-26T01:08:26.513 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.513 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.513 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.513 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399085+0000", 2024-04-26T01:08:26.513 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.492e-06 2024-04-26T01:08:26.513 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.513 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.513 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.513 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399091+0000", 2024-04-26T01:08:26.514 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6780000000000003e-06 2024-04-26T01:08:26.514 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.514 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.514 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.514 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399096+0000", 2024-04-26T01:08:26.514 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9080000000000003e-06 2024-04-26T01:08:26.514 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.514 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.514 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.514 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.401097+0000", 2024-04-26T01:08:26.514 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0020008719999999999 2024-04-26T01:08:26.514 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.514 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.514 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:26.514 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.401190+0000", 2024-04-26T01:08:26.515 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2440000000000003e-05 2024-04-26T01:08:26.515 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.515 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.515 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:26.515 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.401938+0000", 2024-04-26T01:08:26.515 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00074833400000000002 2024-04-26T01:08:26.515 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.515 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.515 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:26.515 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.406495+0000", 2024-04-26T01:08:26.515 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0045574099999999996 2024-04-26T01:08:26.515 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.515 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.515 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:26.516 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.409843+0000", 2024-04-26T01:08:26.516 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033479970000000001 2024-04-26T01:08:26.516 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.516 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.516 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.516 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.409965+0000", 2024-04-26T01:08:26.516 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000121514 2024-04-26T01:08:26.516 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.516 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.516 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.516 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.516 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.516 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:615 3.4s0 3:2d34bde3:::smithi16428812-29:head [append 0~0] snapc f=[f,d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-26T01:08:26.516 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:25.399140+0000", 2024-04-26T01:08:26.517 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.020285112, 2024-04-26T01:08:26.517 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011121401, 2024-04-26T01:08:26.517 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.517 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.517 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:26.517 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:26.517 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:26.517 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:26.517 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 615 2024-04-26T01:08:26.517 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.517 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.517 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.517 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.518 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399140+0000", 2024-04-26T01:08:26.518 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.518 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.518 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.518 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.518 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399140+0000", 2024-04-26T01:08:26.518 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.518 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.518 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.518 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.519 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399140+0000", 2024-04-26T01:08:26.519 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4200000000000001e-07 2024-04-26T01:08:26.519 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.519 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.519 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.519 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399143+0000", 2024-04-26T01:08:26.519 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.604e-06 2024-04-26T01:08:26.519 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.519 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.519 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.519 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399149+0000", 2024-04-26T01:08:26.519 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3450000000000002e-06 2024-04-26T01:08:26.520 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.520 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.520 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.520 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399155+0000", 2024-04-26T01:08:26.520 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9800000000000003e-06 2024-04-26T01:08:26.520 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.520 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.520 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.520 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.402924+0000", 2024-04-26T01:08:26.520 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0037682430000000001 2024-04-26T01:08:26.520 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.521 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.521 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:26.521 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.403216+0000", 2024-04-26T01:08:26.521 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00029238099999999998 2024-04-26T01:08:26.521 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.521 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.521 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:26.521 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.404019+0000", 2024-04-26T01:08:26.521 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00080257600000000005 2024-04-26T01:08:26.521 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.521 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.521 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:26.521 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.407799+0000", 2024-04-26T01:08:26.521 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0037806530000000001 2024-04-26T01:08:26.521 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.522 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.522 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:26.522 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.410124+0000", 2024-04-26T01:08:26.522 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0023246429999999999 2024-04-26T01:08:26.522 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.522 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.522 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.522 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.410261+0000", 2024-04-26T01:08:26.522 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000137534 2024-04-26T01:08:26.522 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.523 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.523 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.523 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.523 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.523 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 59/30 ECSubWriteReply(tid=56, last_complete=59'40, committed=1, applied=1))", 2024-04-26T01:08:26.523 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:25.412873+0000", 2024-04-26T01:08:26.523 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.006552114, 2024-04-26T01:08:26.523 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.43333044199999998, 2024-04-26T01:08:26.523 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.523 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.523 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:26.523 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.523 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.523 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.524 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.412873+0000", 2024-04-26T01:08:26.524 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.524 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.524 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.524 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.524 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.412873+0000", 2024-04-26T01:08:26.524 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.524 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.524 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.524 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.524 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.412874+0000", 2024-04-26T01:08:26.524 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3600000000000003e-07 2024-04-26T01:08:26.524 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.524 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.525 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.525 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.412881+0000", 2024-04-26T01:08:26.525 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3699999999999997e-06 2024-04-26T01:08:26.525 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.525 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.525 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.525 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.846041+0000", 2024-04-26T01:08:26.525 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.43316028400000001 2024-04-26T01:08:26.525 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.525 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.525 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.525 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.846075+0000", 2024-04-26T01:08:26.525 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3213000000000002e-05 2024-04-26T01:08:26.525 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.526 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.526 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.526 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.846162+0000", 2024-04-26T01:08:26.526 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7017999999999999e-05 2024-04-26T01:08:26.526 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.526 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.526 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:26.526 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.846203+0000", 2024-04-26T01:08:26.526 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1921000000000001e-05 2024-04-26T01:08:26.526 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.526 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.526 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.526 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:26.527 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:26.527 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:26.527 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:08:26.529 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:08:26.529 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:26.524+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:26.629 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:26.628+0000 7ff2871de640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 32810) UID: 0 2024-04-26T01:08:26.691 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:26.691 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:26.691 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-04-26T01:08:26.691 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:26.707 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:08:26.724 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:26.725 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:26.730 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:26.728+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:26.750 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:26.750 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:26.777 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:26.778 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:26.806 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:26.807 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:26.824 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:26.825 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:26.825 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-04-26T01:08:26.825 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-04-26T01:08:26.825 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:26.831 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:26.831 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:26.831 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:26.832+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:26.840 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:08:26.859 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:26.860 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:26.888 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:26.888 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:26.920 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:26.920 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:26.932 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:26.928+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:26.948 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:26.948 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:26.980 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:26.981 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:26.995 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:26.995 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:26.995 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:26.996 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:26.996 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.996 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 33/30 ECSubWrite(tid=0, reqid=client.4521.0:22, at_version=33'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:26.996 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.310594+0000", 2024-04-26T01:08:26.996 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 27.682892160000002, 2024-04-26T01:08:26.996 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035400710000000001, 2024-04-26T01:08:26.996 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:26.996 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:26.996 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:26.996 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:26.996 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.997 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:26.997 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310594+0000", 2024-04-26T01:08:26.997 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.997 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.997 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.997 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:26.997 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310594+0000", 2024-04-26T01:08:26.997 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:26.997 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.997 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.997 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:26.998 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310594+0000", 2024-04-26T01:08:26.998 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7500000000000001e-07 2024-04-26T01:08:26.998 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.998 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.998 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:26.998 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310673+0000", 2024-04-26T01:08:26.998 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8784999999999999e-05 2024-04-26T01:08:26.998 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.998 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.998 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:26.998 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310747+0000", 2024-04-26T01:08:26.999 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3899999999999994e-05 2024-04-26T01:08:26.999 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.999 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.999 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:26.999 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310763+0000", 2024-04-26T01:08:26.999 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5769000000000001e-05 2024-04-26T01:08:26.999 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:26.999 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:26.999 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:26.999 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310858+0000", 2024-04-26T01:08:26.999 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.5383000000000005e-05 2024-04-26T01:08:26.999 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.000 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.000 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.000 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310871+0000", 2024-04-26T01:08:27.000 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2887e-05 2024-04-26T01:08:27.000 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.000 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.000 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.000 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314085+0000", 2024-04-26T01:08:27.000 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032136249999999999 2024-04-26T01:08:27.000 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.000 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.000 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.001 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314134+0000", 2024-04-26T01:08:27.001 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9447000000000003e-05 2024-04-26T01:08:27.001 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.001 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.001 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.001 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.001 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.001 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 33/30 ECSubWrite(tid=1, reqid=client.4521.0:23, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:27.001 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.311455+0000", 2024-04-26T01:08:27.001 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 27.682031443, 2024-04-26T01:08:27.001 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027249739999999998, 2024-04-26T01:08:27.001 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.002 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.002 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.002 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.002 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.002 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.002 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311455+0000", 2024-04-26T01:08:27.002 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.002 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.002 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.002 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.002 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311455+0000", 2024-04-26T01:08:27.002 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.003 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.003 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.003 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.003 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311456+0000", 2024-04-26T01:08:27.003 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.012e-06 2024-04-26T01:08:27.003 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.003 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.003 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.003 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311479+0000", 2024-04-26T01:08:27.003 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3456e-05 2024-04-26T01:08:27.003 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.003 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.004 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.004 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311527+0000", 2024-04-26T01:08:27.004 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7827999999999997e-05 2024-04-26T01:08:27.004 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.004 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.004 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.004 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311540+0000", 2024-04-26T01:08:27.004 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2921e-05 2024-04-26T01:08:27.004 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.004 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.004 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.004 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311641+0000", 2024-04-26T01:08:27.005 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000101034 2024-04-26T01:08:27.005 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.005 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.005 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.005 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.311654+0000", 2024-04-26T01:08:27.005 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2781e-05 2024-04-26T01:08:27.005 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.005 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.005 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.005 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314144+0000", 2024-04-26T01:08:27.005 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024907200000000001 2024-04-26T01:08:27.005 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.006 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.006 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.006 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314180+0000", 2024-04-26T01:08:27.006 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.5222000000000003e-05 2024-04-26T01:08:27.006 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.006 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.006 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.006 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.006 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.006 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 32/30 ECSubWrite(tid=1, reqid=client.4521.0:25, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:27.006 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.312642+0000", 2024-04-26T01:08:27.006 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 27.680843931999998, 2024-04-26T01:08:27.006 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032273459999999999, 2024-04-26T01:08:27.007 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.007 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.007 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.007 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.007 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.007 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.007 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312642+0000", 2024-04-26T01:08:27.007 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.007 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.007 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.007 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.007 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312642+0000", 2024-04-26T01:08:27.008 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.008 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.008 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.008 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.008 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312644+0000", 2024-04-26T01:08:27.008 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2720000000000001e-06 2024-04-26T01:08:27.008 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.008 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.008 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.008 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312878+0000", 2024-04-26T01:08:27.008 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00023409 2024-04-26T01:08:27.008 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.008 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.009 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.009 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312930+0000", 2024-04-26T01:08:27.009 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2799000000000001e-05 2024-04-26T01:08:27.009 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.009 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.009 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.009 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312944+0000", 2024-04-26T01:08:27.009 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3687e-05 2024-04-26T01:08:27.009 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.009 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.009 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.009 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.313045+0000", 2024-04-26T01:08:27.010 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000100782 2024-04-26T01:08:27.010 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.010 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.010 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.010 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.313058+0000", 2024-04-26T01:08:27.010 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2785999999999999e-05 2024-04-26T01:08:27.010 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.010 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.010 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.010 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.315782+0000", 2024-04-26T01:08:27.010 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027242439999999998 2024-04-26T01:08:27.010 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.010 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.011 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.011 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.315870+0000", 2024-04-26T01:08:27.011 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7686e-05 2024-04-26T01:08:27.011 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.011 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.011 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.011 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.011 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.011 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 37/30 ECSubWrite(tid=17, reqid=client.4521.0:243, at_version=37'13, trim_to=0'0, roll_forward_to=36'11))", 2024-04-26T01:08:27.011 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.423152+0000", 2024-04-26T01:08:27.011 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 23.570334621000001, 2024-04-26T01:08:27.011 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027083010000000002, 2024-04-26T01:08:27.012 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.012 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.012 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.012 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.012 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.012 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.012 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423152+0000", 2024-04-26T01:08:27.012 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.012 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.012 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.012 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.012 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423152+0000", 2024-04-26T01:08:27.013 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.013 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.013 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.013 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.013 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423152+0000", 2024-04-26T01:08:27.013 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8900000000000001e-07 2024-04-26T01:08:27.013 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.013 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.013 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.013 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423668+0000", 2024-04-26T01:08:27.013 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00051661399999999998 2024-04-26T01:08:27.013 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.014 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.014 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.014 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423705+0000", 2024-04-26T01:08:27.014 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6186999999999998e-05 2024-04-26T01:08:27.014 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.014 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.014 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.014 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423714+0000", 2024-04-26T01:08:27.014 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0969999999999995e-06 2024-04-26T01:08:27.014 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.014 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.015 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.015 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423786+0000", 2024-04-26T01:08:27.015 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2013000000000003e-05 2024-04-26T01:08:27.015 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.015 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.015 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.015 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.423796+0000", 2024-04-26T01:08:27.015 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0179e-05 2024-04-26T01:08:27.015 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.015 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.015 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.015 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.425806+0000", 2024-04-26T01:08:27.015 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0020104570000000002 2024-04-26T01:08:27.016 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.016 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.016 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.016 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.425860+0000", 2024-04-26T01:08:27.016 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3565e-05 2024-04-26T01:08:27.016 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.016 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.016 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.016 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.016 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.016 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 38/30 ECSubWrite(tid=27, reqid=osd.2.0:27, at_version=38'21, trim_to=0'0, roll_forward_to=37'19))", 2024-04-26T01:08:27.016 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.359904+0000", 2024-04-26T01:08:27.017 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 22.633581832000001, 2024-04-26T01:08:27.017 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0030231030000000001, 2024-04-26T01:08:27.017 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.017 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.017 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.017 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.017 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.017 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.017 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.359904+0000", 2024-04-26T01:08:27.017 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.017 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.017 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.018 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.018 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.359904+0000", 2024-04-26T01:08:27.018 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.018 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.018 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.018 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.018 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.359905+0000", 2024-04-26T01:08:27.018 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1399999999999997e-07 2024-04-26T01:08:27.018 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.018 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.018 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.018 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.359945+0000", 2024-04-26T01:08:27.019 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0318999999999998e-05 2024-04-26T01:08:27.019 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.019 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.019 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.019 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.360030+0000", 2024-04-26T01:08:27.019 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5159000000000002e-05 2024-04-26T01:08:27.019 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.019 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.019 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.019 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.360050+0000", 2024-04-26T01:08:27.019 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9412999999999999e-05 2024-04-26T01:08:27.019 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.019 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.020 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.020 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.360211+0000", 2024-04-26T01:08:27.020 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000161652 2024-04-26T01:08:27.020 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.020 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.020 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.020 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.360242+0000", 2024-04-26T01:08:27.020 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0522999999999999e-05 2024-04-26T01:08:27.020 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.020 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.020 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.021 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.362859+0000", 2024-04-26T01:08:27.021 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0026173099999999999 2024-04-26T01:08:27.021 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.021 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.021 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.021 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.362927+0000", 2024-04-26T01:08:27.021 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8312999999999995e-05 2024-04-26T01:08:27.021 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.021 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.021 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.021 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.021 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.021 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 38/30 ECSubWrite(tid=28, reqid=osd.2.0:28, at_version=38'23, trim_to=0'0, roll_forward_to=37'19))", 2024-04-26T01:08:27.022 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.361206+0000", 2024-04-26T01:08:27.022 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 22.6322805, 2024-04-26T01:08:27.022 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027194950000000002, 2024-04-26T01:08:27.022 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.022 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.022 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.022 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.022 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.022 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.022 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.361206+0000", 2024-04-26T01:08:27.022 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.023 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.023 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.023 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.023 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.361206+0000", 2024-04-26T01:08:27.023 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.023 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.023 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.023 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.023 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.361207+0000", 2024-04-26T01:08:27.023 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4529999999999999e-06 2024-04-26T01:08:27.023 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.023 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.024 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.024 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.361246+0000", 2024-04-26T01:08:27.024 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9246000000000003e-05 2024-04-26T01:08:27.024 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.024 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.024 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.024 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.361339+0000", 2024-04-26T01:08:27.024 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2659999999999997e-05 2024-04-26T01:08:27.024 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.024 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.024 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.024 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.361362+0000", 2024-04-26T01:08:27.025 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2946e-05 2024-04-26T01:08:27.025 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.025 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.025 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.025 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.361951+0000", 2024-04-26T01:08:27.025 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00058876399999999998 2024-04-26T01:08:27.025 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.025 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.025 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.025 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.361995+0000", 2024-04-26T01:08:27.025 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4100000000000001e-05 2024-04-26T01:08:27.025 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.025 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.026 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.026 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.363851+0000", 2024-04-26T01:08:27.026 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0018566500000000001 2024-04-26T01:08:27.026 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.026 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.026 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.026 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.363925+0000", 2024-04-26T01:08:27.026 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3676000000000004e-05 2024-04-26T01:08:27.026 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.026 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.026 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.027 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.027 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.027 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 38/30 ECSubWrite(tid=72, reqid=client.4521.0:263, at_version=38'22, trim_to=0'0, roll_forward_to=36'21))", 2024-04-26T01:08:27.027 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.408150+0000", 2024-04-26T01:08:27.027 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 22.585336666, 2024-04-26T01:08:27.027 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032528909999999999, 2024-04-26T01:08:27.027 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.027 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.027 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.027 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.027 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.027 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.027 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.408150+0000", 2024-04-26T01:08:27.028 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.028 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.028 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.028 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.028 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.408150+0000", 2024-04-26T01:08:27.028 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.028 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.028 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.028 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.028 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.408150+0000", 2024-04-26T01:08:27.028 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6399999999999998e-07 2024-04-26T01:08:27.029 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.029 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.029 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.029 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.408742+0000", 2024-04-26T01:08:27.029 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00059185499999999996 2024-04-26T01:08:27.029 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.029 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.029 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.029 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.408823+0000", 2024-04-26T01:08:27.029 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0949e-05 2024-04-26T01:08:27.029 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.029 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.029 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.030 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.408841+0000", 2024-04-26T01:08:27.030 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7876e-05 2024-04-26T01:08:27.030 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.030 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.030 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.030 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.409004+0000", 2024-04-26T01:08:27.030 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016316200000000001 2024-04-26T01:08:27.030 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.030 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.030 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.030 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.409031+0000", 2024-04-26T01:08:27.031 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7160000000000001e-05 2024-04-26T01:08:27.031 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.031 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.031 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.031 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.411310+0000", 2024-04-26T01:08:27.031 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022787329999999998 2024-04-26T01:08:27.031 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.031 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.031 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.031 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.411402+0000", 2024-04-26T01:08:27.031 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2792000000000002e-05 2024-04-26T01:08:27.031 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.032 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.032 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.032 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.032 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.032 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 38/30 ECSubWrite(tid=74, reqid=client.4521.0:264, at_version=38'23, trim_to=0'0, roll_forward_to=38'22))", 2024-04-26T01:08:27.032 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.415518+0000", 2024-04-26T01:08:27.032 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 22.577967741999998, 2024-04-26T01:08:27.032 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032675970000000001, 2024-04-26T01:08:27.032 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.032 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.032 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.032 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.033 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.033 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.033 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.415518+0000", 2024-04-26T01:08:27.033 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.033 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.033 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.033 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.033 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.415518+0000", 2024-04-26T01:08:27.033 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.033 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.033 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.033 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.034 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.415520+0000", 2024-04-26T01:08:27.034 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6270000000000001e-06 2024-04-26T01:08:27.034 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.034 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.034 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.034 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.415852+0000", 2024-04-26T01:08:27.034 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00033187000000000001 2024-04-26T01:08:27.034 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.034 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.034 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.034 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.416299+0000", 2024-04-26T01:08:27.034 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00044752600000000001 2024-04-26T01:08:27.035 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.035 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.035 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.035 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.416325+0000", 2024-04-26T01:08:27.035 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5259999999999999e-05 2024-04-26T01:08:27.035 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.035 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.035 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.035 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.416639+0000", 2024-04-26T01:08:27.035 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00031457399999999997 2024-04-26T01:08:27.035 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.035 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.036 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.036 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.416663+0000", 2024-04-26T01:08:27.036 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3583000000000001e-05 2024-04-26T01:08:27.036 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.036 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.036 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.036 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.418708+0000", 2024-04-26T01:08:27.036 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0020455550000000001 2024-04-26T01:08:27.036 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.036 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.036 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.036 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.418786+0000", 2024-04-26T01:08:27.036 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7601999999999993e-05 2024-04-26T01:08:27.037 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.037 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.037 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.037 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.037 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.037 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 38/30 ECSubWrite(tid=77, reqid=client.4521.0:265, at_version=0'0, trim_to=0'0, roll_forward_to=38'24))", 2024-04-26T01:08:27.037 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.425003+0000", 2024-04-26T01:08:27.037 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 22.568483662999999, 2024-04-26T01:08:27.037 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0028818540000000001, 2024-04-26T01:08:27.037 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.037 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.038 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.038 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.038 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.038 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.038 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.425003+0000", 2024-04-26T01:08:27.038 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.038 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.038 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.038 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.038 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.425003+0000", 2024-04-26T01:08:27.038 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.038 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.039 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.039 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.039 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.425004+0000", 2024-04-26T01:08:27.039 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9360000000000002e-06 2024-04-26T01:08:27.039 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.039 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.039 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.039 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.425028+0000", 2024-04-26T01:08:27.039 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.385e-05 2024-04-26T01:08:27.039 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.039 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.040 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.040 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.425121+0000", 2024-04-26T01:08:27.040 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2843000000000005e-05 2024-04-26T01:08:27.040 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.040 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.040 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.040 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.425146+0000", 2024-04-26T01:08:27.040 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4490000000000001e-05 2024-04-26T01:08:27.040 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.040 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.040 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.040 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.425342+0000", 2024-04-26T01:08:27.041 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000196021 2024-04-26T01:08:27.041 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.041 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.041 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.041 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.425378+0000", 2024-04-26T01:08:27.041 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6223000000000002e-05 2024-04-26T01:08:27.041 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.041 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.041 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.041 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.427829+0000", 2024-04-26T01:08:27.041 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002451183 2024-04-26T01:08:27.041 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.041 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.042 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.042 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.427884+0000", 2024-04-26T01:08:27.042 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5308e-05 2024-04-26T01:08:27.042 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.042 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.042 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.042 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.042 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.042 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs3 50/30 ECSubWrite(tid=214, reqid=osd.4.0:214, at_version=50'22, trim_to=0'0, roll_forward_to=45'20))", 2024-04-26T01:08:27.042 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:16.046662+0000", 2024-04-26T01:08:27.042 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 10.946824205, 2024-04-26T01:08:27.043 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0043439849999999999, 2024-04-26T01:08:27.043 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.043 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.043 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.043 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.043 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.043 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.043 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046662+0000", 2024-04-26T01:08:27.043 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.043 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.043 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.043 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.044 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046662+0000", 2024-04-26T01:08:27.044 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.044 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.044 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.044 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.044 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046662+0000", 2024-04-26T01:08:27.044 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6500000000000001e-07 2024-04-26T01:08:27.044 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.044 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.044 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.044 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046688+0000", 2024-04-26T01:08:27.044 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6217e-05 2024-04-26T01:08:27.045 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.045 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.045 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.045 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046803+0000", 2024-04-26T01:08:27.045 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000114837 2024-04-26T01:08:27.045 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.045 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.045 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.045 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046812+0000", 2024-04-26T01:08:27.045 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2560000000000008e-06 2024-04-26T01:08:27.045 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.045 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.046 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.046 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046879+0000", 2024-04-26T01:08:27.046 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6812999999999999e-05 2024-04-26T01:08:27.046 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.046 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.046 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.046 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046890+0000", 2024-04-26T01:08:27.046 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0842999999999999e-05 2024-04-26T01:08:27.046 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.046 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.046 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.046 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.050916+0000", 2024-04-26T01:08:27.047 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0040258079999999996 2024-04-26T01:08:27.047 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.047 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.047 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.047 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051006+0000", 2024-04-26T01:08:27.047 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0045999999999998e-05 2024-04-26T01:08:27.047 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.047 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.047 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.047 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.047 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.047 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.6s4 50/30 ECSubWrite(tid=41, reqid=osd.0.0:41, at_version=50'24, trim_to=0'0, roll_forward_to=45'22))", 2024-04-26T01:08:27.048 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:16.051314+0000", 2024-04-26T01:08:27.048 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 10.942172516999999, 2024-04-26T01:08:27.048 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003465143, 2024-04-26T01:08:27.048 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.048 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.048 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.048 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.048 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.048 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.048 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051314+0000", 2024-04-26T01:08:27.048 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.048 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.049 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.049 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.049 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051314+0000", 2024-04-26T01:08:27.049 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.049 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.049 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.049 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.049 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051314+0000", 2024-04-26T01:08:27.049 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0800000000000001e-07 2024-04-26T01:08:27.049 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.049 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.049 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.049 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051361+0000", 2024-04-26T01:08:27.050 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6925000000000002e-05 2024-04-26T01:08:27.050 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.050 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.050 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.050 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051412+0000", 2024-04-26T01:08:27.050 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1161e-05 2024-04-26T01:08:27.050 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.050 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.050 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.050 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051421+0000", 2024-04-26T01:08:27.050 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2839999999999995e-06 2024-04-26T01:08:27.050 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.051 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.051 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.051 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051864+0000", 2024-04-26T01:08:27.051 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00044309800000000001 2024-04-26T01:08:27.051 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.051 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.051 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.051 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051893+0000", 2024-04-26T01:08:27.051 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8782999999999999e-05 2024-04-26T01:08:27.051 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.051 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.051 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.052 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.054700+0000", 2024-04-26T01:08:27.052 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0028072549999999998 2024-04-26T01:08:27.052 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.052 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.052 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.052 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.054779+0000", 2024-04-26T01:08:27.052 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8429000000000004e-05 2024-04-26T01:08:27.052 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.052 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.052 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.052 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.052 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.052 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs3 55/30 ECSubWrite(tid=252, reqid=client.4521.0:548, at_version=55'28, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:27.053 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:21.083658+0000", 2024-04-26T01:08:27.053 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 5.9098281230000005, 2024-04-26T01:08:27.053 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0087585849999999993, 2024-04-26T01:08:27.053 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.053 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.053 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.053 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.053 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.053 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.053 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.083658+0000", 2024-04-26T01:08:27.053 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.054 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.054 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.054 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.054 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.083658+0000", 2024-04-26T01:08:27.054 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.054 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.054 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.054 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.054 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.083659+0000", 2024-04-26T01:08:27.054 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3300000000000002e-07 2024-04-26T01:08:27.054 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.054 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.055 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.055 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.083943+0000", 2024-04-26T01:08:27.055 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00028484799999999998 2024-04-26T01:08:27.055 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.055 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.055 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.055 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.084081+0000", 2024-04-26T01:08:27.055 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000137818 2024-04-26T01:08:27.055 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.055 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.055 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.055 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.084108+0000", 2024-04-26T01:08:27.056 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6287000000000001e-05 2024-04-26T01:08:27.056 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.056 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.056 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.056 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.084277+0000", 2024-04-26T01:08:27.056 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016978800000000001 2024-04-26T01:08:27.056 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.056 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.056 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.056 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.084304+0000", 2024-04-26T01:08:27.056 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6967000000000001e-05 2024-04-26T01:08:27.056 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.056 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.057 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.057 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.092229+0000", 2024-04-26T01:08:27.057 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0079250229999999998 2024-04-26T01:08:27.057 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.057 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.057 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.057 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.092417+0000", 2024-04-26T01:08:27.057 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000187321 2024-04-26T01:08:27.057 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.057 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.057 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.057 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.057 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.058 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs3 55/30 ECSubWrite(tid=254, reqid=client.4521.0:549, at_version=55'29, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:27.058 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:21.087618+0000", 2024-04-26T01:08:27.058 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 5.9058677910000004, 2024-04-26T01:08:27.058 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0099810469999999998, 2024-04-26T01:08:27.058 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.058 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.058 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.058 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.058 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.058 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.058 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.087618+0000", 2024-04-26T01:08:27.058 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.059 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.059 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.059 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.059 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.087618+0000", 2024-04-26T01:08:27.059 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.059 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.059 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.059 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.059 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.087620+0000", 2024-04-26T01:08:27.059 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.86e-06 2024-04-26T01:08:27.059 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.059 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.060 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.060 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.087838+0000", 2024-04-26T01:08:27.060 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000217741 2024-04-26T01:08:27.060 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.060 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.060 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.060 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.087932+0000", 2024-04-26T01:08:27.060 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4392e-05 2024-04-26T01:08:27.060 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.060 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.060 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.060 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.087960+0000", 2024-04-26T01:08:27.061 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7376999999999999e-05 2024-04-26T01:08:27.061 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.061 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.061 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.061 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.092546+0000", 2024-04-26T01:08:27.061 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0045858519999999996 2024-04-26T01:08:27.061 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.061 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.061 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.061 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.092570+0000", 2024-04-26T01:08:27.061 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3920000000000001e-05 2024-04-26T01:08:27.061 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.061 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.062 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.062 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.097506+0000", 2024-04-26T01:08:27.062 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.004936686 2024-04-26T01:08:27.062 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.062 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.062 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.062 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.097599+0000", 2024-04-26T01:08:27.062 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3219000000000003e-05 2024-04-26T01:08:27.062 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.062 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.062 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.063 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.063 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.063 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 55/44 ECSubWrite(tid=55, reqid=osd.2.0:55, at_version=55'44, trim_to=0'0, roll_forward_to=52'42))", 2024-04-26T01:08:27.063 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:21.094514+0000", 2024-04-26T01:08:27.063 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 5.8989724670000001, 2024-04-26T01:08:27.063 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035312939999999999, 2024-04-26T01:08:27.063 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.063 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.063 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.063 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.063 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.063 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.064 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094514+0000", 2024-04-26T01:08:27.064 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.064 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.064 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.064 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.064 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094514+0000", 2024-04-26T01:08:27.064 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.064 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.064 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.064 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.064 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094514+0000", 2024-04-26T01:08:27.064 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.37e-07 2024-04-26T01:08:27.064 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.065 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.065 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.065 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094571+0000", 2024-04-26T01:08:27.065 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6693000000000001e-05 2024-04-26T01:08:27.065 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.065 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.065 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.065 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094688+0000", 2024-04-26T01:08:27.065 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011739300000000001 2024-04-26T01:08:27.065 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.065 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.065 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.066 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094712+0000", 2024-04-26T01:08:27.066 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3108999999999998e-05 2024-04-26T01:08:27.066 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.066 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.066 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.066 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094824+0000", 2024-04-26T01:08:27.066 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011225 2024-04-26T01:08:27.066 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.066 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.066 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.066 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094838+0000", 2024-04-26T01:08:27.066 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4361e-05 2024-04-26T01:08:27.067 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.067 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.067 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.067 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.097990+0000", 2024-04-26T01:08:27.067 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0031517070000000001 2024-04-26T01:08:27.067 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.067 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.067 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.067 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.098045+0000", 2024-04-26T01:08:27.067 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5143999999999999e-05 2024-04-26T01:08:27.067 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.067 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.068 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.068 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.068 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.068 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.6s4 55/30 ECSubWrite(tid=49, reqid=osd.0.0:49, at_version=55'31, trim_to=0'0, roll_forward_to=52'29))", 2024-04-26T01:08:27.068 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:21.098290+0000", 2024-04-26T01:08:27.068 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 5.895196093, 2024-04-26T01:08:27.068 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002774491, 2024-04-26T01:08:27.068 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.068 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.068 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.068 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.068 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.069 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.069 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.098290+0000", 2024-04-26T01:08:27.069 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.069 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.069 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.069 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.069 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.098290+0000", 2024-04-26T01:08:27.069 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.069 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.069 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.069 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.069 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.098290+0000", 2024-04-26T01:08:27.070 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0899999999999997e-07 2024-04-26T01:08:27.070 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.070 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.070 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.070 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.098336+0000", 2024-04-26T01:08:27.070 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5494000000000002e-05 2024-04-26T01:08:27.070 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.070 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.070 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.070 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.098406+0000", 2024-04-26T01:08:27.070 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9943000000000004e-05 2024-04-26T01:08:27.071 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.071 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.071 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.071 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.098418+0000", 2024-04-26T01:08:27.071 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2075e-05 2024-04-26T01:08:27.071 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.071 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.071 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.071 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.098522+0000", 2024-04-26T01:08:27.071 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000104404 2024-04-26T01:08:27.071 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.071 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.072 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.072 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.098537+0000", 2024-04-26T01:08:27.072 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4355999999999999e-05 2024-04-26T01:08:27.072 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.072 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.072 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.072 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.101025+0000", 2024-04-26T01:08:27.072 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024885649999999999 2024-04-26T01:08:27.072 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.072 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.072 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.073 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.101065+0000", 2024-04-26T01:08:27.073 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9344999999999997e-05 2024-04-26T01:08:27.073 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.073 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.073 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.073 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.073 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.073 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs3 55/30 ECSubWrite(tid=256, reqid=client.4521.0:550, at_version=55'30, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:27.073 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:21.102027+0000", 2024-04-26T01:08:27.073 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 5.8914595470000002, 2024-04-26T01:08:27.073 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0037147669999999999, 2024-04-26T01:08:27.073 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.074 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.074 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.074 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.074 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.074 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.074 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102027+0000", 2024-04-26T01:08:27.074 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.074 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.074 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.074 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.074 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102027+0000", 2024-04-26T01:08:27.075 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.075 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.075 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102028+0000", 2024-04-26T01:08:27.075 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.703e-06 2024-04-26T01:08:27.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.075 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.075 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102149+0000", 2024-04-26T01:08:27.075 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000121149 2024-04-26T01:08:27.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.076 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.076 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.076 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102279+0000", 2024-04-26T01:08:27.076 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012956299999999999 2024-04-26T01:08:27.076 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.076 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.076 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.076 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102305+0000", 2024-04-26T01:08:27.076 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.552e-05 2024-04-26T01:08:27.076 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.076 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.076 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102458+0000", 2024-04-26T01:08:27.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015376100000000001 2024-04-26T01:08:27.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102485+0000", 2024-04-26T01:08:27.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6529999999999998e-05 2024-04-26T01:08:27.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.105647+0000", 2024-04-26T01:08:27.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003162045 2024-04-26T01:08:27.078 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.078 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.078 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.078 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.105741+0000", 2024-04-26T01:08:27.078 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4495999999999998e-05 2024-04-26T01:08:27.078 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.078 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.078 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.078 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.078 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.078 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs3 59/30 ECSubWrite(tid=331, reqid=client.4521.0:616, at_version=59'47, trim_to=0'0, roll_forward_to=59'45))", 2024-04-26T01:08:27.078 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:25.463449+0000", 2024-04-26T01:08:27.079 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.5300367669999999, 2024-04-26T01:08:27.079 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0038682870000000002, 2024-04-26T01:08:27.079 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.079 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.079 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.079 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.079 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.079 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.079 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.463449+0000", 2024-04-26T01:08:27.079 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.079 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.079 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.080 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.080 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.463449+0000", 2024-04-26T01:08:27.080 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.080 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.080 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.080 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.080 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.463452+0000", 2024-04-26T01:08:27.080 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5600000000000001e-06 2024-04-26T01:08:27.080 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.080 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.080 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.080 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.463935+0000", 2024-04-26T01:08:27.081 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00048299299999999997 2024-04-26T01:08:27.081 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.081 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.081 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.081 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.464046+0000", 2024-04-26T01:08:27.081 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011133000000000001 2024-04-26T01:08:27.081 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.081 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.081 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.081 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.464071+0000", 2024-04-26T01:08:27.081 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4502999999999999e-05 2024-04-26T01:08:27.081 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.082 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.082 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.082 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.464240+0000", 2024-04-26T01:08:27.082 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016889800000000001 2024-04-26T01:08:27.082 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.082 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.082 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.082 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.464268+0000", 2024-04-26T01:08:27.082 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7886999999999999e-05 2024-04-26T01:08:27.082 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.082 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.082 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.083 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.467212+0000", 2024-04-26T01:08:27.083 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0029448640000000002 2024-04-26T01:08:27.083 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.083 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.083 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.083 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.467318+0000", 2024-04-26T01:08:27.083 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000105252 2024-04-26T01:08:27.083 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.083 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.083 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.083 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.084 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.084 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs3 59/30 ECSubWrite(tid=334, reqid=client.4521.0:618, at_version=59'49, trim_to=0'0, roll_forward_to=59'48))", 2024-04-26T01:08:27.084 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:25.474538+0000", 2024-04-26T01:08:27.084 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.5189480369999999, 2024-04-26T01:08:27.084 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0026155229999999998, 2024-04-26T01:08:27.084 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.084 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.084 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.084 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.084 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.084 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.084 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.474538+0000", 2024-04-26T01:08:27.085 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.085 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.085 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.085 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.085 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.474538+0000", 2024-04-26T01:08:27.085 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.085 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.085 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.085 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.085 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.474540+0000", 2024-04-26T01:08:27.085 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.61e-06 2024-04-26T01:08:27.086 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.086 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.086 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.086 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.474874+0000", 2024-04-26T01:08:27.086 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00033376 2024-04-26T01:08:27.086 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.086 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.086 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.086 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.475089+0000", 2024-04-26T01:08:27.086 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00021547499999999999 2024-04-26T01:08:27.086 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.086 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.087 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.087 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.475112+0000", 2024-04-26T01:08:27.087 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2629999999999998e-05 2024-04-26T01:08:27.087 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.087 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.087 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.087 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.475260+0000", 2024-04-26T01:08:27.087 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014877 2024-04-26T01:08:27.087 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.087 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.087 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.087 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.475283+0000", 2024-04-26T01:08:27.088 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3102999999999999e-05 2024-04-26T01:08:27.088 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.088 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.088 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.088 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.477120+0000", 2024-04-26T01:08:27.088 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001836826 2024-04-26T01:08:27.088 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.088 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.088 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.088 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.477154+0000", 2024-04-26T01:08:27.088 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3349000000000002e-05 2024-04-26T01:08:27.088 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.088 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.089 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.089 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.089 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.089 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.6s4 60/30 ECSubWrite(tid=59, reqid=client.4521.0:641, at_version=60'39, trim_to=0'0, roll_forward_to=58'37))", 2024-04-26T01:08:27.089 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:26.174498+0000", 2024-04-26T01:08:27.089 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.81898857400000002, 2024-04-26T01:08:27.089 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003094865, 2024-04-26T01:08:27.089 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.089 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.089 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.089 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.089 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.089 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.090 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174498+0000", 2024-04-26T01:08:27.090 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.090 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.090 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.090 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.090 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174498+0000", 2024-04-26T01:08:27.090 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.090 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.090 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.090 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.090 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174498+0000", 2024-04-26T01:08:27.090 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5300000000000004e-07 2024-04-26T01:08:27.091 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.091 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.091 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.091 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174537+0000", 2024-04-26T01:08:27.091 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8435e-05 2024-04-26T01:08:27.091 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.091 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.091 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.091 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174618+0000", 2024-04-26T01:08:27.091 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1435999999999998e-05 2024-04-26T01:08:27.091 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.091 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.092 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.092 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174632+0000", 2024-04-26T01:08:27.092 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3919e-05 2024-04-26T01:08:27.092 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.092 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.092 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.092 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174719+0000", 2024-04-26T01:08:27.092 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7492999999999997e-05 2024-04-26T01:08:27.092 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.092 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.092 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.092 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174734+0000", 2024-04-26T01:08:27.092 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.455e-05 2024-04-26T01:08:27.093 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.093 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.093 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.093 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.177532+0000", 2024-04-26T01:08:27.093 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027984720000000002 2024-04-26T01:08:27.093 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.093 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.093 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.093 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.177592+0000", 2024-04-26T01:08:27.093 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0007000000000003e-05 2024-04-26T01:08:27.093 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.093 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.094 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.094 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.094 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.094 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs3 60/30 ECSubWrite(tid=351, reqid=osd.4.0:351, at_version=60'54, trim_to=0'0, roll_forward_to=59'50))", 2024-04-26T01:08:27.094 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:26.178228+0000", 2024-04-26T01:08:27.094 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.81525864999999997, 2024-04-26T01:08:27.094 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003424889, 2024-04-26T01:08:27.094 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:27.094 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:27.094 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:27.094 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:27.094 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.095 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:27.095 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.178228+0000", 2024-04-26T01:08:27.095 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.095 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.095 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.095 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:27.095 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.178228+0000", 2024-04-26T01:08:27.095 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:27.095 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.095 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.095 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:27.095 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.178228+0000", 2024-04-26T01:08:27.095 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9399999999999995e-07 2024-04-26T01:08:27.096 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.096 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.096 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:27.096 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.178254+0000", 2024-04-26T01:08:27.096 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5741e-05 2024-04-26T01:08:27.096 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.096 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.096 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:27.096 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.178376+0000", 2024-04-26T01:08:27.096 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000122063 2024-04-26T01:08:27.096 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.096 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.096 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:27.097 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.178388+0000", 2024-04-26T01:08:27.097 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1905999999999999e-05 2024-04-26T01:08:27.097 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.097 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.097 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:27.097 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.179887+0000", 2024-04-26T01:08:27.097 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0014988029999999999 2024-04-26T01:08:27.097 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.097 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.097 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:27.097 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.179900+0000", 2024-04-26T01:08:27.097 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3042e-05 2024-04-26T01:08:27.098 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.098 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.098 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:27.098 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.181602+0000", 2024-04-26T01:08:27.098 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0017023190000000001 2024-04-26T01:08:27.098 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:27.098 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:27.098 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:27.098 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.181652+0000", 2024-04-26T01:08:27.098 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.0521e-05 2024-04-26T01:08:27.098 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.098 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.098 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.099 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:27.099 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:27.099 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:27.101 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:08:27.101 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:27.101 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:27.102 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:08:27.102 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-04-26T01:08:27.102 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:27.103 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:27.103 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:27.096+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:27.106 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:27.106 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:27.141 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:27.142 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:27.173 INFO:tasks.rados.rados.0.smithi164.stdout:249: done (0 left) 2024-04-26T01:08:27.173 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 32 current snap is 14 2024-04-26T01:08:27.173 INFO:tasks.rados.rados.0.smithi164.stdout:250: seq_num 117 ranges {0=425984,425984=606208,1032192=540672,1572864=409600} 2024-04-26T01:08:27.174 INFO:tasks.rados.rados.0.smithi164.stdout:250: writing smithi16428812-32 from 0 to 425984 tid 1 2024-04-26T01:08:27.175 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:27.175 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:27.178 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-04-26T01:08:27.178 INFO:tasks.rados.rados.0.smithi164.stdout:250: writing smithi16428812-32 from 425984 to 1032192 tid 2 2024-04-26T01:08:27.182 INFO:tasks.rados.rados.0.smithi164.stdout:250: writing smithi16428812-32 from 1032192 to 1572864 tid 3 2024-04-26T01:08:27.184 INFO:tasks.rados.rados.0.smithi164.stdout:250: writing smithi16428812-32 from 1572864 to 1982464 tid 4 2024-04-26T01:08:27.184 INFO:tasks.rados.rados.0.smithi164.stdout:251: copy_from oid 17 from oid 5 current snap is 14 2024-04-26T01:08:27.184 INFO:tasks.rados.rados.0.smithi164.stdout:252: snap_create 2024-04-26T01:08:27.195 INFO:tasks.rados.rados.0.smithi164.stdout:250: finishing write tid 1 to smithi16428812-32 2024-04-26T01:08:27.198 INFO:tasks.rados.rados.0.smithi164.stdout:251: finishing copy_from racing read to smithi16428812-17 2024-04-26T01:08:27.200 INFO:tasks.rados.rados.0.smithi164.stdout:250: finishing write tid 2 to smithi16428812-32 2024-04-26T01:08:27.200 INFO:tasks.rados.rados.0.smithi164.stdout:250: finishing write tid 3 to smithi16428812-32 2024-04-26T01:08:27.200 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:27.200+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:27.203 INFO:tasks.rados.rados.0.smithi164.stdout:250: finishing write tid 4 to smithi16428812-32 2024-04-26T01:08:27.203 INFO:tasks.rados.rados.0.smithi164.stdout:251: finishing copy_from to smithi16428812-17 2024-04-26T01:08:27.203 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 17 v 18 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-26T01:08:27.203 INFO:tasks.rados.rados.0.smithi164.stdout:250: finishing write tid 5 to smithi16428812-32 2024-04-26T01:08:27.206 INFO:tasks.rados.rados.0.smithi164.stdout:250: finishing write tid 6 to smithi16428812-32 2024-04-26T01:08:27.206 INFO:tasks.rados.rados.0.smithi164.stdout:250: oid 32 updating version 0 to 45 2024-04-26T01:08:27.206 INFO:tasks.rados.rados.0.smithi164.stdout:250: oid 32 updating version 45 to 46 2024-04-26T01:08:27.206 INFO:tasks.rados.rados.0.smithi164.stdout:250: oid 32 updating version 46 to 47 2024-04-26T01:08:27.206 INFO:tasks.rados.rados.0.smithi164.stdout:250: oid 32 updating version 47 to 52 2024-04-26T01:08:27.206 INFO:tasks.rados.rados.0.smithi164.stdout:250: oid 32 updating version 52 to 53 2024-04-26T01:08:27.206 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 32 v 53 (ObjNum 117 snap 14 seq_num 117) dirty exists 2024-04-26T01:08:27.206 INFO:tasks.rados.rados.0.smithi164.stdout:250: left oid 32 (ObjNum 117 snap 14 seq_num 117) 2024-04-26T01:08:27.247 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:27.248 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:27.262 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:08:27.302 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:27.300+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:27.386 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:27.386 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:27.400 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:08:27.402 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:27.399+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:27.503 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:27.504+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:27.520 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:27.520 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:27.534 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:08:27.604 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:27.600+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:27.662 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:27.662 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:27.676 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:08:27.705 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:27.700+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:27.795 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:27.796 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:27.806 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:27.804+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:27.810 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:08:27.907 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:27.904+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:27.937 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:27.937 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:27.951 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:08:28.007 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:28.005+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:28.077 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:28.077 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:28.091 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:08:28.108 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:28.108+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:28.180 INFO:tasks.rados.rados.0.smithi164.stdout:250: done (2 left) 2024-04-26T01:08:28.180 INFO:tasks.rados.rados.0.smithi164.stdout:251: done (1 left) 2024-04-26T01:08:28.180 INFO:tasks.rados.rados.0.smithi164.stdout:252: done (0 left) 2024-04-26T01:08:28.180 INFO:tasks.rados.rados.0.smithi164.stdout:253: copy_from oid 3 from oid 40 current snap is 15 2024-04-26T01:08:28.180 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 15 current snap is 15 2024-04-26T01:08:28.180 INFO:tasks.rados.rados.0.smithi164.stdout:254: seq_num 118 ranges {3162112=442368,3604480=753664,4358144=655360,5013504=147456} 2024-04-26T01:08:28.209 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:28.207+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:28.237 INFO:tasks.rados.rados.0.smithi164.stdout:254: writing smithi16428812-15 from 3162112 to 3604480 tid 1 2024-04-26T01:08:28.246 INFO:tasks.rados.rados.0.smithi164.stdout:254: writing smithi16428812-15 from 3604480 to 4358144 tid 2 2024-04-26T01:08:28.254 INFO:tasks.rados.rados.0.smithi164.stdout:254: writing smithi16428812-15 from 4358144 to 5013504 tid 3 2024-04-26T01:08:28.255 INFO:tasks.rados.rados.0.smithi164.stdout:254: writing smithi16428812-15 from 5013504 to 5160960 tid 4 2024-04-26T01:08:28.255 INFO:tasks.rados.rados.0.smithi164.stdout:255: read oid 6 snap -1 2024-04-26T01:08:28.256 INFO:tasks.rados.rados.0.smithi164.stdout:255: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-26T01:08:28.256 INFO:tasks.rados.rados.0.smithi164.stdout:256: snap_remove snap 14 2024-04-26T01:08:28.275 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:28.275 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:28.290 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:08:28.310 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:28.312+0000 7ff2871de640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 32810) UID: 0 2024-04-26T01:08:28.411 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:28.412+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:28.416 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:28.416 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:28.431 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:08:28.511 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:28.511+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:28.550 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:28.550 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:28.565 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:08:28.612 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:28.609+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:28.690 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:28.690 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:28.704 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:08:28.748 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:28.745+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:28.814 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:28.812+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:28.874 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:28.874 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:28.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:08:28.936 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:28.935+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:29.016 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:29.016+0000 7ff2871de640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 32810) UID: 0 2024-04-26T01:08:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:29.034 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:08:29.117 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:29.112+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:29.189 INFO:tasks.rados.rados.0.smithi164.stdout:256: done (3 left) 2024-04-26T01:08:29.189 INFO:tasks.rados.rados.0.smithi164.stdout:257: read oid 22 snap -1 2024-04-26T01:08:29.189 INFO:tasks.rados.rados.0.smithi164.stdout:257: expect (ObjNum 95 snap 8 seq_num 95) 2024-04-26T01:08:29.189 INFO:tasks.rados.rados.0.smithi164.stdout:258: read oid 14 snap -1 2024-04-26T01:08:29.189 INFO:tasks.rados.rados.0.smithi164.stdout:258: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:08:29.189 INFO:tasks.rados.rados.0.smithi164.stdout:259: read oid 8 snap -1 2024-04-26T01:08:29.189 INFO:tasks.rados.rados.0.smithi164.stdout:259: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-26T01:08:29.189 INFO:tasks.rados.rados.0.smithi164.stdout:260: delete oid 25 current snap is 15 2024-04-26T01:08:29.189 INFO:tasks.rados.rados.0.smithi164.stdout:253: finishing copy_from racing read to smithi16428812-3 2024-04-26T01:08:29.189 INFO:tasks.rados.rados.0.smithi164.stdout:253: finishing copy_from to smithi16428812-3 2024-04-26T01:08:29.189 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 3 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-26T01:08:29.190 INFO:tasks.rados.rados.0.smithi164.stdout:254: finishing write tid 1 to smithi16428812-15 2024-04-26T01:08:29.190 INFO:tasks.rados.rados.0.smithi164.stdout:254: finishing write tid 2 to smithi16428812-15 2024-04-26T01:08:29.190 INFO:tasks.rados.rados.0.smithi164.stdout:254: finishing write tid 3 to smithi16428812-15 2024-04-26T01:08:29.190 INFO:tasks.rados.rados.0.smithi164.stdout:254: finishing write tid 4 to smithi16428812-15 2024-04-26T01:08:29.190 INFO:tasks.rados.rados.0.smithi164.stdout:254: finishing write tid 5 to smithi16428812-15 2024-04-26T01:08:29.190 INFO:tasks.rados.rados.0.smithi164.stdout:255: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-26T01:08:29.196 INFO:tasks.rados.rados.0.smithi164.stdout:254: finishing write tid 6 to smithi16428812-15 2024-04-26T01:08:29.196 INFO:tasks.rados.rados.0.smithi164.stdout:254: oid 15 updating version 0 to 47 2024-04-26T01:08:29.197 INFO:tasks.rados.rados.0.smithi164.stdout:254: oid 15 version 47 is already newer than 44 2024-04-26T01:08:29.197 INFO:tasks.rados.rados.0.smithi164.stdout:254: oid 15 version 47 is already newer than 43 2024-04-26T01:08:29.197 INFO:tasks.rados.rados.0.smithi164.stdout:254: oid 15 version 47 is already newer than 46 2024-04-26T01:08:29.197 INFO:tasks.rados.rados.0.smithi164.stdout:254: oid 15 version 47 is already newer than 45 2024-04-26T01:08:29.197 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 15 v 47 (ObjNum 118 snap 15 seq_num 118) dirty exists 2024-04-26T01:08:29.197 INFO:tasks.rados.rados.0.smithi164.stdout:254: left oid 15 (ObjNum 118 snap 15 seq_num 118) 2024-04-26T01:08:29.200 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:29.200 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:29.202 INFO:tasks.rados.rados.0.smithi164.stdout:259: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-26T01:08:29.205 INFO:tasks.rados.rados.0.smithi164.stdout:258: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:08:29.218 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:29.220+0000 7ff2871de640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 32810) UID: 0 2024-04-26T01:08:29.219 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:08:29.219 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:08:29.220 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-26T01:08:29.319 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:29.313+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:29.336 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:29.336 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:29.351 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:08:29.389 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:29.389 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:29.414 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:29.414 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:29.419 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:29.416+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:29.453 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:29.453 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:29.474 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:29.474 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:29.488 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:08:29.493 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:29.493 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:29.520 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:29.516+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:29.532 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:29.532 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:29.572 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:29.572 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:29.602 INFO:tasks.rados.rados.0.smithi164.stdout:257: expect (ObjNum 95 snap 8 seq_num 95) 2024-04-26T01:08:29.612 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:29.612 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:29.613 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:29.613 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:29.613 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:08:29.613 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:08:29.613 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:29.621 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:29.619+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:29.627 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:08:29.638 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:29.638 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:29.662 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:29.662 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:29.688 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:29.688 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:29.714 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:29.714 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:29.716 INFO:tasks.rados.rados.0.smithi164.stdout:253: done (6 left) 2024-04-26T01:08:29.716 INFO:tasks.rados.rados.0.smithi164.stdout:254: done (5 left) 2024-04-26T01:08:29.716 INFO:tasks.rados.rados.0.smithi164.stdout:255: done (4 left) 2024-04-26T01:08:29.716 INFO:tasks.rados.rados.0.smithi164.stdout:257: done (3 left) 2024-04-26T01:08:29.716 INFO:tasks.rados.rados.0.smithi164.stdout:258: done (2 left) 2024-04-26T01:08:29.716 INFO:tasks.rados.rados.0.smithi164.stdout:259: done (1 left) 2024-04-26T01:08:29.716 INFO:tasks.rados.rados.0.smithi164.stdout:260: done (0 left) 2024-04-26T01:08:29.716 INFO:tasks.rados.rados.0.smithi164.stdout:261: rollback oid 17 current snap is 15 2024-04-26T01:08:29.716 INFO:tasks.rados.rados.0.smithi164.stdout:261: done (0 left) 2024-04-26T01:08:29.716 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 37 current snap is 15 2024-04-26T01:08:29.717 INFO:tasks.rados.rados.0.smithi164.stdout:262: seq_num 120 ranges {1474560=475136,1949696=655360,2605056=622592,3227648=278528} 2024-04-26T01:08:29.722 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:29.724+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:29.739 INFO:tasks.rados.rados.0.smithi164.stdout:262: writing smithi16428812-37 from 1474560 to 1949696 tid 1 2024-04-26T01:08:29.747 INFO:tasks.rados.rados.0.smithi164.stdout:262: writing smithi16428812-37 from 1949696 to 2605056 tid 2 2024-04-26T01:08:29.755 INFO:tasks.rados.rados.0.smithi164.stdout:262: writing smithi16428812-37 from 2605056 to 3227648 tid 3 2024-04-26T01:08:29.758 INFO:tasks.rados.rados.0.smithi164.stdout:262: writing smithi16428812-37 from 3227648 to 3506176 tid 4 2024-04-26T01:08:29.758 INFO:tasks.rados.rados.0.smithi164.stdout:263: rollback oid 20 current snap is 15 2024-04-26T01:08:29.758 INFO:tasks.rados.rados.0.smithi164.stdout:263: done (1 left) 2024-04-26T01:08:29.758 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 1 current snap is 15 2024-04-26T01:08:29.759 INFO:tasks.rados.rados.0.smithi164.stdout:264: seq_num 121 ranges {0=491520,491520=802816,1294336=720896,2015232=32768} 2024-04-26T01:08:29.764 INFO:tasks.rados.rados.0.smithi164.stdout:264: writing smithi16428812-1 from 0 to 491520 tid 1 2024-04-26T01:08:29.774 INFO:tasks.rados.rados.0.smithi164.stdout:264: writing smithi16428812-1 from 491520 to 1294336 tid 2 2024-04-26T01:08:29.781 INFO:tasks.rados.rados.0.smithi164.stdout:264: writing smithi16428812-1 from 1294336 to 2015232 tid 3 2024-04-26T01:08:29.782 INFO:tasks.rados.rados.0.smithi164.stdout:264: writing smithi16428812-1 from 2015232 to 2048000 tid 4 2024-04-26T01:08:29.782 INFO:tasks.rados.rados.0.smithi164.stdout:265: rmattr oid 14 current snap is 15 2024-04-26T01:08:29.782 INFO:tasks.rados.rados.0.smithi164.stdout:265: done (2 left) 2024-04-26T01:08:29.782 INFO:tasks.rados.rados.0.smithi164.stdout:266: read oid 13 snap -1 2024-04-26T01:08:29.782 INFO:tasks.rados.rados.0.smithi164.stdout:266: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-26T01:08:29.783 INFO:tasks.rados.rados.0.smithi164.stdout:267: snap_create 2024-04-26T01:08:29.783 INFO:tasks.rados.rados.0.smithi164.stdout:262: finishing write tid 1 to smithi16428812-37 2024-04-26T01:08:29.783 INFO:tasks.rados.rados.0.smithi164.stdout:262: finishing write tid 2 to smithi16428812-37 2024-04-26T01:08:29.783 INFO:tasks.rados.rados.0.smithi164.stdout:262: finishing write tid 3 to smithi16428812-37 2024-04-26T01:08:29.783 INFO:tasks.rados.rados.0.smithi164.stdout:262: finishing write tid 4 to smithi16428812-37 2024-04-26T01:08:29.783 INFO:tasks.rados.rados.0.smithi164.stdout:262: finishing write tid 5 to smithi16428812-37 2024-04-26T01:08:29.783 INFO:tasks.rados.rados.0.smithi164.stdout:262: finishing write tid 6 to smithi16428812-37 2024-04-26T01:08:29.783 INFO:tasks.rados.rados.0.smithi164.stdout:262: oid 37 updating version 0 to 16 2024-04-26T01:08:29.783 INFO:tasks.rados.rados.0.smithi164.stdout:262: oid 37 updating version 16 to 17 2024-04-26T01:08:29.783 INFO:tasks.rados.rados.0.smithi164.stdout:262: oid 37 updating version 17 to 18 2024-04-26T01:08:29.783 INFO:tasks.rados.rados.0.smithi164.stdout:262: oid 37 version 18 is already newer than 15 2024-04-26T01:08:29.783 INFO:tasks.rados.rados.0.smithi164.stdout:262: oid 37 version 18 is already newer than 14 2024-04-26T01:08:29.783 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 37 v 18 (ObjNum 120 snap 15 seq_num 120) dirty exists 2024-04-26T01:08:29.784 INFO:tasks.rados.rados.0.smithi164.stdout:262: left oid 37 (ObjNum 120 snap 15 seq_num 120) 2024-04-26T01:08:29.784 INFO:tasks.rados.rados.0.smithi164.stdout:264: finishing write tid 1 to smithi16428812-1 2024-04-26T01:08:29.784 INFO:tasks.rados.rados.0.smithi164.stdout:264: finishing write tid 2 to smithi16428812-1 2024-04-26T01:08:29.791 INFO:tasks.rados.rados.0.smithi164.stdout:264: finishing write tid 3 to smithi16428812-1 2024-04-26T01:08:29.799 INFO:tasks.rados.rados.0.smithi164.stdout:264: finishing write tid 4 to smithi16428812-1 2024-04-26T01:08:29.799 INFO:tasks.rados.rados.0.smithi164.stdout:264: finishing write tid 5 to smithi16428812-1 2024-04-26T01:08:29.802 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:29.802 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:08:29.802 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:08:29.802 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:08:29.802 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.802 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 36/30 ECSubWrite(tid=15, reqid=client.4521.0:233, at_version=36'11, trim_to=0'0, roll_forward_to=33'9))", 2024-04-26T01:08:29.802 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:02.480147+0000", 2024-04-26T01:08:29.803 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 27.320232238999999, 2024-04-26T01:08:29.803 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.64949222799999995, 2024-04-26T01:08:29.803 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.803 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.803 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.803 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.803 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.803 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.803 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480147+0000", 2024-04-26T01:08:29.803 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.803 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.803 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.803 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.803 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480147+0000", 2024-04-26T01:08:29.803 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.804 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.804 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.804 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480148+0000", 2024-04-26T01:08:29.804 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2300000000000001e-06 2024-04-26T01:08:29.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.804 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.804 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.804 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480517+0000", 2024-04-26T01:08:29.804 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00036900700000000003 2024-04-26T01:08:29.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.804 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.804 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.805 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125262+0000", 2024-04-26T01:08:29.805 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.64474460600000005 2024-04-26T01:08:29.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.805 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.805 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.805 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125278+0000", 2024-04-26T01:08:29.805 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5622999999999999e-05 2024-04-26T01:08:29.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.805 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.805 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.805 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125483+0000", 2024-04-26T01:08:29.805 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000205105 2024-04-26T01:08:29.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.806 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.806 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.806 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125502+0000", 2024-04-26T01:08:29.806 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8876999999999999e-05 2024-04-26T01:08:29.806 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.806 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.806 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.806 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.129565+0000", 2024-04-26T01:08:29.806 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0040633930000000002 2024-04-26T01:08:29.806 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.806 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.806 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.807 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.129639+0000", 2024-04-26T01:08:29.807 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4387000000000005e-05 2024-04-26T01:08:29.807 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.807 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.807 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.807 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.807 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.807 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.5s6 43/30 ECSubWrite(tid=35, reqid=osd.15.0:35, at_version=43'30, trim_to=0'0, roll_forward_to=42'28))", 2024-04-26T01:08:29.807 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:29.807 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 20.393820499, 2024-04-26T01:08:29.807 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011310863000000001, 2024-04-26T01:08:29.807 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.807 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.807 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2e-07 2024-04-26T01:08:29.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406612+0000", 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2206000000000001e-05 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406709+0000", 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7106000000000002e-05 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406737+0000", 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8629999999999999e-05 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.809 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.810 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.810 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.407351+0000", 2024-04-26T01:08:29.810 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00061410800000000004 2024-04-26T01:08:29.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.810 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.810 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.810 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.407377+0000", 2024-04-26T01:08:29.810 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5057000000000001e-05 2024-04-26T01:08:29.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.810 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.810 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.810 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.417771+0000", 2024-04-26T01:08:29.810 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.01039402 2024-04-26T01:08:29.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.811 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.811 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.811 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.417870+0000", 2024-04-26T01:08:29.811 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9215999999999993e-05 2024-04-26T01:08:29.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.811 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.811 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.811 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 50/44 ECSubWrite(tid=51, reqid=osd.2.0:51, at_version=50'40, trim_to=0'0, roll_forward_to=45'38))", 2024-04-26T01:08:29.811 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.058137+0000", 2024-04-26T01:08:29.811 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.742242117, 2024-04-26T01:08:29.811 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0092019839999999999, 2024-04-26T01:08:29.812 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.812 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.812 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.812 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.812 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.812 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.812 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058137+0000", 2024-04-26T01:08:29.812 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.812 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.812 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.812 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.812 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058137+0000", 2024-04-26T01:08:29.812 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.812 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.813 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.813 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.813 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058138+0000", 2024-04-26T01:08:29.813 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.3300000000000001e-07 2024-04-26T01:08:29.813 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.813 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.813 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.813 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058197+0000", 2024-04-26T01:08:29.813 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9110000000000002e-05 2024-04-26T01:08:29.813 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.813 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.813 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.813 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058307+0000", 2024-04-26T01:08:29.814 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000109706 2024-04-26T01:08:29.814 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.814 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.814 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.814 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058330+0000", 2024-04-26T01:08:29.814 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3540000000000002e-05 2024-04-26T01:08:29.814 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.814 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.814 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.814 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058521+0000", 2024-04-26T01:08:29.814 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019083500000000001 2024-04-26T01:08:29.814 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.814 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.815 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.815 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058548+0000", 2024-04-26T01:08:29.815 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6950000000000001e-05 2024-04-26T01:08:29.815 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.815 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.815 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.815 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.067237+0000", 2024-04-26T01:08:29.815 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0086888769999999994 2024-04-26T01:08:29.815 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.815 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.815 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.815 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.067339+0000", 2024-04-26T01:08:29.815 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010213300000000001 2024-04-26T01:08:29.816 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.816 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.816 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.816 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.816 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.816 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs4 50/44 ECSubWrite(tid=22, reqid=osd.9.0:22, at_version=50'37, trim_to=0'0, roll_forward_to=47'35))", 2024-04-26T01:08:29.816 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:08:29.816 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.741170798000001, 2024-04-26T01:08:29.816 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016916468, 2024-04-26T01:08:29.816 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.816 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.816 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.816 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.817 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.817 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.817 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:08:29.817 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.817 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.817 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.817 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.817 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:08:29.817 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.817 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.817 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.817 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.817 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:08:29.818 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9299999999999996e-07 2024-04-26T01:08:29.818 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.818 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.818 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.818 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059282+0000", 2024-04-26T01:08:29.818 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2019000000000003e-05 2024-04-26T01:08:29.818 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.818 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.818 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.818 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059397+0000", 2024-04-26T01:08:29.818 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000115569 2024-04-26T01:08:29.818 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.818 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.818 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059419+0000", 2024-04-26T01:08:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1497000000000001e-05 2024-04-26T01:08:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064015+0000", 2024-04-26T01:08:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0045966059999999996 2024-04-26T01:08:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064041+0000", 2024-04-26T01:08:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5570000000000001e-05 2024-04-26T01:08:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.075972+0000", 2024-04-26T01:08:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011930802000000001 2024-04-26T01:08:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076125+0000", 2024-04-26T01:08:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015351199999999999 2024-04-26T01:08:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 50/44 ECSubWrite(tid=215, reqid=client.4521.0:490, at_version=50'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-26T01:08:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.059258+0000", 2024-04-26T01:08:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.741120926000001, 2024-04-26T01:08:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014073085000000001, 2024-04-26T01:08:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059258+0000", 2024-04-26T01:08:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059258+0000", 2024-04-26T01:08:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059261+0000", 2024-04-26T01:08:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3829999999999999e-06 2024-04-26T01:08:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.826 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.826 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.826 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059926+0000", 2024-04-26T01:08:29.826 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00066483199999999997 2024-04-26T01:08:29.826 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.826 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.826 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.826 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060011+0000", 2024-04-26T01:08:29.826 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5248999999999996e-05 2024-04-26T01:08:29.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.827 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.827 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.827 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060033+0000", 2024-04-26T01:08:29.827 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1566999999999999e-05 2024-04-26T01:08:29.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.827 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.827 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.827 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060387+0000", 2024-04-26T01:08:29.827 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000354793 2024-04-26T01:08:29.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.827 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.827 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.827 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060449+0000", 2024-04-26T01:08:29.827 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.1957000000000003e-05 2024-04-26T01:08:29.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.828 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.828 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.828 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073174+0000", 2024-04-26T01:08:29.828 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012724998 2024-04-26T01:08:29.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.828 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.828 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.828 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073332+0000", 2024-04-26T01:08:29.828 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015730599999999999 2024-04-26T01:08:29.828 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.828 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.828 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.829 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.829 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.0s5 50/30 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=50'32, trim_to=0'0, roll_forward_to=45'30))", 2024-04-26T01:08:29.829 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.062205+0000", 2024-04-26T01:08:29.829 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.738174546, 2024-04-26T01:08:29.829 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.01120789, 2024-04-26T01:08:29.829 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.829 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.829 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.829 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.829 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.829 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.829 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062205+0000", 2024-04-26T01:08:29.829 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.829 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.829 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.830 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.830 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062205+0000", 2024-04-26T01:08:29.830 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.830 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.830 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.830 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.830 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062207+0000", 2024-04-26T01:08:29.830 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.844e-06 2024-04-26T01:08:29.830 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.830 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.830 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.831 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062258+0000", 2024-04-26T01:08:29.831 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.1388999999999999e-05 2024-04-26T01:08:29.831 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.831 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.831 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.831 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062376+0000", 2024-04-26T01:08:29.831 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00011767199999999999 2024-04-26T01:08:29.831 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.831 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.831 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.831 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062399+0000", 2024-04-26T01:08:29.831 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3410000000000001e-05 2024-04-26T01:08:29.831 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.831 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.831 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.832 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062583+0000", 2024-04-26T01:08:29.832 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00018336900000000001 2024-04-26T01:08:29.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.832 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.832 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.832 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062611+0000", 2024-04-26T01:08:29.832 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8189999999999999e-05 2024-04-26T01:08:29.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.832 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.832 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.832 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073316+0000", 2024-04-26T01:08:29.832 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010704824 2024-04-26T01:08:29.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.833 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.833 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.833 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073413+0000", 2024-04-26T01:08:29.833 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7192000000000001e-05 2024-04-26T01:08:29.833 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.833 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.833 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.833 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.833 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.2s3 50/44 ECSubWrite(tid=212, reqid=osd.13.0:212, at_version=50'24, trim_to=0'0, roll_forward_to=45'22))", 2024-04-26T01:08:29.833 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.063661+0000", 2024-04-26T01:08:29.833 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.736717928000001, 2024-04-26T01:08:29.833 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012570563999999999, 2024-04-26T01:08:29.833 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.834 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.834 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.834 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.834 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.834 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.834 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063661+0000", 2024-04-26T01:08:29.834 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.834 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.834 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.834 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063661+0000", 2024-04-26T01:08:29.834 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.835 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.835 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.835 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063662+0000", 2024-04-26T01:08:29.835 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.37e-07 2024-04-26T01:08:29.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.835 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.835 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.835 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063721+0000", 2024-04-26T01:08:29.835 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.8696999999999997e-05 2024-04-26T01:08:29.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.835 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.835 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.835 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063860+0000", 2024-04-26T01:08:29.835 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013883799999999999 2024-04-26T01:08:29.836 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.836 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.836 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.836 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063882+0000", 2024-04-26T01:08:29.836 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2313000000000001e-05 2024-04-26T01:08:29.836 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.836 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.836 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.836 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064661+0000", 2024-04-26T01:08:29.836 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00077955699999999995 2024-04-26T01:08:29.836 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.836 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.836 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.836 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064686+0000", 2024-04-26T01:08:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4984e-05 2024-04-26T01:08:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.075967+0000", 2024-04-26T01:08:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011280646 2024-04-26T01:08:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076232+0000", 2024-04-26T01:08:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00026499200000000003 2024-04-26T01:08:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.838 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 50/30 ECSubWrite(tid=216, reqid=osd.4.0:214, at_version=0'0, trim_to=0'0, roll_forward_to=50'22))", 2024-04-26T01:08:29.838 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.065107+0000", 2024-04-26T01:08:29.838 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.735272026000001, 2024-04-26T01:08:29.838 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0087328609999999997, 2024-04-26T01:08:29.838 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.838 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.838 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.838 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.838 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.838 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065107+0000", 2024-04-26T01:08:29.838 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.839 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.839 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.839 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065107+0000", 2024-04-26T01:08:29.839 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.839 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.839 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.839 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065108+0000", 2024-04-26T01:08:29.839 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.0999999999999997e-07 2024-04-26T01:08:29.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.839 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.839 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.839 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065125+0000", 2024-04-26T01:08:29.840 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7010000000000001e-05 2024-04-26T01:08:29.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.840 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.840 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.840 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065360+0000", 2024-04-26T01:08:29.840 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00023507199999999999 2024-04-26T01:08:29.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.840 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.840 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.840 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065382+0000", 2024-04-26T01:08:29.840 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1993000000000001e-05 2024-04-26T01:08:29.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.840 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.840 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065672+0000", 2024-04-26T01:08:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00028928100000000001 2024-04-26T01:08:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065696+0000", 2024-04-26T01:08:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4273e-05 2024-04-26T01:08:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073748+0000", 2024-04-26T01:08:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0080520490000000004 2024-04-26T01:08:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.842 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073840+0000", 2024-04-26T01:08:29.842 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2373000000000005e-05 2024-04-26T01:08:29.842 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.842 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.842 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.842 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.842 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.842 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 50/44 ECSubWrite(tid=218, reqid=osd.4.0:218, at_version=50'34, trim_to=0'0, roll_forward_to=47'30))", 2024-04-26T01:08:29.842 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.065939+0000", 2024-04-26T01:08:29.842 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.734440412, 2024-04-26T01:08:29.842 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010065406000000001, 2024-04-26T01:08:29.842 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.842 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.843 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.843 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.843 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.843 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.843 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065939+0000", 2024-04-26T01:08:29.843 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.843 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.843 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.843 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065939+0000", 2024-04-26T01:08:29.843 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.843 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.844 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.844 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065941+0000", 2024-04-26T01:08:29.844 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9199999999999998e-06 2024-04-26T01:08:29.844 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.844 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.844 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.844 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065994+0000", 2024-04-26T01:08:29.844 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3439000000000001e-05 2024-04-26T01:08:29.844 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.845 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.845 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.845 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066088+0000", 2024-04-26T01:08:29.845 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.365e-05 2024-04-26T01:08:29.845 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.845 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.845 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.845 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066110+0000", 2024-04-26T01:08:29.845 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1912999999999998e-05 2024-04-26T01:08:29.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.846 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.846 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.846 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066779+0000", 2024-04-26T01:08:29.846 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00066918399999999997 2024-04-26T01:08:29.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.846 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.846 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.846 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066810+0000", 2024-04-26T01:08:29.846 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0446999999999998e-05 2024-04-26T01:08:29.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.847 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.847 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.847 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.075902+0000", 2024-04-26T01:08:29.847 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0090923529999999992 2024-04-26T01:08:29.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.847 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.847 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.847 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076004+0000", 2024-04-26T01:08:29.847 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0001025 2024-04-26T01:08:29.847 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.847 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.847 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.847 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.848 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.8s6 50/30 ECSubWrite(tid=42, reqid=osd.5.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=50'26))", 2024-04-26T01:08:29.848 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.068230+0000", 2024-04-26T01:08:29.848 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.732149561, 2024-04-26T01:08:29.848 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0080202260000000001, 2024-04-26T01:08:29.848 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.848 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.848 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.848 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.848 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.848 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.848 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068230+0000", 2024-04-26T01:08:29.848 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.849 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.849 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068230+0000", 2024-04-26T01:08:29.849 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.849 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.849 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068232+0000", 2024-04-26T01:08:29.849 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.804e-06 2024-04-26T01:08:29.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.849 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.849 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068261+0000", 2024-04-26T01:08:29.850 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9595999999999999e-05 2024-04-26T01:08:29.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.850 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.850 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068341+0000", 2024-04-26T01:08:29.850 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9302999999999996e-05 2024-04-26T01:08:29.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.850 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.850 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068355+0000", 2024-04-26T01:08:29.850 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.465e-05 2024-04-26T01:08:29.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.851 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.851 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068495+0000", 2024-04-26T01:08:29.851 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013948500000000001 2024-04-26T01:08:29.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.851 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.851 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068520+0000", 2024-04-26T01:08:29.851 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5015999999999999e-05 2024-04-26T01:08:29.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.851 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.851 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076177+0000", 2024-04-26T01:08:29.851 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0076572489999999997 2024-04-26T01:08:29.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.852 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.852 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076250+0000", 2024-04-26T01:08:29.852 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.3122999999999998e-05 2024-04-26T01:08:29.852 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.852 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.852 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.852 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 55/44 ECSubWrite(tid=253, reqid=osd.4.0:253, at_version=55'43, trim_to=0'0, roll_forward_to=52'41))", 2024-04-26T01:08:29.852 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:21.084283+0000", 2024-04-26T01:08:29.852 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 8.7160967300000003, 2024-04-26T01:08:29.853 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061481820000000003, 2024-04-26T01:08:29.853 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.853 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.853 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.853 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.853 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.853 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084283+0000", 2024-04-26T01:08:29.853 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.853 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.853 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084283+0000", 2024-04-26T01:08:29.854 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.854 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.854 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084283+0000", 2024-04-26T01:08:29.854 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0699999999999997e-07 2024-04-26T01:08:29.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.854 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.854 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084362+0000", 2024-04-26T01:08:29.854 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9083000000000002e-05 2024-04-26T01:08:29.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.855 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.855 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084566+0000", 2024-04-26T01:08:29.855 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00020390800000000001 2024-04-26T01:08:29.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.855 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.855 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084589+0000", 2024-04-26T01:08:29.855 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3213e-05 2024-04-26T01:08:29.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.855 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.855 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084751+0000", 2024-04-26T01:08:29.855 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016103600000000001 2024-04-26T01:08:29.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.856 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.856 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084781+0000", 2024-04-26T01:08:29.856 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0372999999999999e-05 2024-04-26T01:08:29.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.856 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.856 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.090337+0000", 2024-04-26T01:08:29.856 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.005556503 2024-04-26T01:08:29.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.856 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.856 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.090431+0000", 2024-04-26T01:08:29.857 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.3459000000000001e-05 2024-04-26T01:08:29.857 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.857 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.857 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.857 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.857 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 55/30 ECSubWrite(tid=252, reqid=client.4521.0:548, at_version=55'28, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:29.857 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:21.084368+0000", 2024-04-26T01:08:29.857 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 8.7160111629999992, 2024-04-26T01:08:29.857 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0068931720000000004, 2024-04-26T01:08:29.857 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.857 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.857 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.858 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.858 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.858 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.858 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084368+0000", 2024-04-26T01:08:29.858 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.858 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.858 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.858 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084368+0000", 2024-04-26T01:08:29.858 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.858 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.859 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.859 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084369+0000", 2024-04-26T01:08:29.859 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.1999999999999999e-07 2024-04-26T01:08:29.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.859 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.859 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.859 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085034+0000", 2024-04-26T01:08:29.859 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00066459099999999997 2024-04-26T01:08:29.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.859 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.859 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.859 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085069+0000", 2024-04-26T01:08:29.859 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4965999999999997e-05 2024-04-26T01:08:29.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.860 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.860 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.860 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085081+0000", 2024-04-26T01:08:29.860 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2422999999999999e-05 2024-04-26T01:08:29.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.860 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.860 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.860 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085243+0000", 2024-04-26T01:08:29.860 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000162466 2024-04-26T01:08:29.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.860 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.861 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.861 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085270+0000", 2024-04-26T01:08:29.861 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6775999999999999e-05 2024-04-26T01:08:29.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.861 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.861 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.861 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091175+0000", 2024-04-26T01:08:29.861 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059045340000000003 2024-04-26T01:08:29.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.861 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.861 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.861 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091261+0000", 2024-04-26T01:08:29.862 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.6695999999999998e-05 2024-04-26T01:08:29.862 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.862 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.862 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.862 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.862 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 55/44 ECSubWrite(tid=55, reqid=osd.2.0:55, at_version=55'44, trim_to=0'0, roll_forward_to=52'42))", 2024-04-26T01:08:29.862 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:21.093675+0000", 2024-04-26T01:08:29.862 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 8.7067042469999993, 2024-04-26T01:08:29.862 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066519099999999996, 2024-04-26T01:08:29.862 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.862 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.863 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.863 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.863 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.863 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093675+0000", 2024-04-26T01:08:29.863 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.863 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.863 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093675+0000", 2024-04-26T01:08:29.863 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.863 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.864 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093676+0000", 2024-04-26T01:08:29.864 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.3e-07 2024-04-26T01:08:29.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.864 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.864 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093734+0000", 2024-04-26T01:08:29.864 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7833e-05 2024-04-26T01:08:29.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.864 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.864 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093836+0000", 2024-04-26T01:08:29.864 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010234900000000001 2024-04-26T01:08:29.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.865 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.865 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093859+0000", 2024-04-26T01:08:29.865 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3383e-05 2024-04-26T01:08:29.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.865 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.865 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.094252+0000", 2024-04-26T01:08:29.865 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000392424 2024-04-26T01:08:29.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.866 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.866 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.094287+0000", 2024-04-26T01:08:29.866 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5713000000000002e-05 2024-04-26T01:08:29.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.866 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.866 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.100231+0000", 2024-04-26T01:08:29.866 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059430519999999999 2024-04-26T01:08:29.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.867 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.867 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.100327+0000", 2024-04-26T01:08:29.867 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6526000000000004e-05 2024-04-26T01:08:29.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.867 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.867 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.867 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 59/30 ECSubWrite(tid=320, reqid=osd.4.0:320, at_version=59'43, trim_to=0'0, roll_forward_to=58'41))", 2024-04-26T01:08:29.867 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.159033+0000", 2024-04-26T01:08:29.867 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.6413465540000001, 2024-04-26T01:08:29.867 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0060271350000000003, 2024-04-26T01:08:29.868 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.868 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.868 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.868 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.868 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.868 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159033+0000", 2024-04-26T01:08:29.868 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.868 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.868 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159033+0000", 2024-04-26T01:08:29.868 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.869 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.869 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159033+0000", 2024-04-26T01:08:29.869 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9699999999999996e-07 2024-04-26T01:08:29.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.869 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.869 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159076+0000", 2024-04-26T01:08:29.869 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.2616e-05 2024-04-26T01:08:29.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.870 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.870 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159156+0000", 2024-04-26T01:08:29.870 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.0400000000000003e-05 2024-04-26T01:08:29.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.870 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.870 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159180+0000", 2024-04-26T01:08:29.870 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3898999999999999e-05 2024-04-26T01:08:29.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.871 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.871 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159352+0000", 2024-04-26T01:08:29.871 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017211599999999999 2024-04-26T01:08:29.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.871 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.871 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159382+0000", 2024-04-26T01:08:29.871 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9410000000000001e-05 2024-04-26T01:08:29.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.871 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.871 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.164958+0000", 2024-04-26T01:08:29.872 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.005575628 2024-04-26T01:08:29.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.872 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.872 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.165060+0000", 2024-04-26T01:08:29.872 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000102469 2024-04-26T01:08:29.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.872 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.873 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 59/30 ECSubWrite(tid=322, reqid=osd.4.0:322, at_version=59'45, trim_to=0'0, roll_forward_to=58'41))", 2024-04-26T01:08:29.873 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.160873+0000", 2024-04-26T01:08:29.873 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.6395067210000001, 2024-04-26T01:08:29.873 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0064776979999999996, 2024-04-26T01:08:29.873 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.873 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.873 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.873 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.873 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.873 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160873+0000", 2024-04-26T01:08:29.873 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.874 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.874 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.874 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160873+0000", 2024-04-26T01:08:29.874 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.874 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.874 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.874 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160874+0000", 2024-04-26T01:08:29.874 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5030000000000001e-06 2024-04-26T01:08:29.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.875 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.875 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.875 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160924+0000", 2024-04-26T01:08:29.875 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0158999999999999e-05 2024-04-26T01:08:29.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.875 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.875 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.875 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161010+0000", 2024-04-26T01:08:29.875 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5285999999999996e-05 2024-04-26T01:08:29.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.875 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.875 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.875 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161032+0000", 2024-04-26T01:08:29.876 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2427000000000001e-05 2024-04-26T01:08:29.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.876 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.876 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161983+0000", 2024-04-26T01:08:29.876 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00095064900000000003 2024-04-26T01:08:29.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.876 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.876 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.162010+0000", 2024-04-26T01:08:29.876 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7786000000000001e-05 2024-04-26T01:08:29.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.877 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.877 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167263+0000", 2024-04-26T01:08:29.877 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0052521979999999996 2024-04-26T01:08:29.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.877 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.877 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167350+0000", 2024-04-26T01:08:29.877 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7689999999999996e-05 2024-04-26T01:08:29.877 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.877 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.877 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.878 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.878 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 60/30 ECSubWrite(tid=349, reqid=osd.4.0:349, at_version=60'52, trim_to=0'0, roll_forward_to=59'50))", 2024-04-26T01:08:29.878 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.179549+0000", 2024-04-26T01:08:29.878 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 3.6208300429999998, 2024-04-26T01:08:29.878 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0055201859999999998, 2024-04-26T01:08:29.878 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.878 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.878 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.878 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.878 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.878 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.878 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.179549+0000", 2024-04-26T01:08:29.878 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.879 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.879 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.879 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.179549+0000", 2024-04-26T01:08:29.879 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.879 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.879 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.879 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.179550+0000", 2024-04-26T01:08:29.879 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.4999999999999998e-07 2024-04-26T01:08:29.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.879 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.879 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.879 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.179583+0000", 2024-04-26T01:08:29.880 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.3525999999999997e-05 2024-04-26T01:08:29.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.880 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.880 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.179833+0000", 2024-04-26T01:08:29.880 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00024941500000000001 2024-04-26T01:08:29.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.880 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.880 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.179850+0000", 2024-04-26T01:08:29.880 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7007000000000002e-05 2024-04-26T01:08:29.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.881 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.881 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.179995+0000", 2024-04-26T01:08:29.881 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014573200000000001 2024-04-26T01:08:29.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.881 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.881 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.881 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.180020+0000", 2024-04-26T01:08:29.881 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4607e-05 2024-04-26T01:08:29.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.881 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.881 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.881 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.184956+0000", 2024-04-26T01:08:29.881 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0049354300000000002 2024-04-26T01:08:29.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.882 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.882 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.185070+0000", 2024-04-26T01:08:29.882 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00011401899999999999 2024-04-26T01:08:29.882 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.882 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.882 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.882 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 60/30 ECSubWrite(tid=351, reqid=osd.4.0:351, at_version=60'54, trim_to=0'0, roll_forward_to=59'50))", 2024-04-26T01:08:29.882 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.179586+0000", 2024-04-26T01:08:29.882 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 3.6207933470000002, 2024-04-26T01:08:29.882 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0055487260000000004, 2024-04-26T01:08:29.883 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.883 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.883 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.883 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.883 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.883 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.179586+0000", 2024-04-26T01:08:29.883 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.884 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.884 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.179586+0000", 2024-04-26T01:08:29.884 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.884 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.884 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.179586+0000", 2024-04-26T01:08:29.884 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4700000000000002e-07 2024-04-26T01:08:29.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.884 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.884 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.179611+0000", 2024-04-26T01:08:29.885 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4263000000000001e-05 2024-04-26T01:08:29.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.885 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.885 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.885 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.179904+0000", 2024-04-26T01:08:29.885 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000292961 2024-04-26T01:08:29.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.885 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.885 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.885 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.179920+0000", 2024-04-26T01:08:29.885 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6623e-05 2024-04-26T01:08:29.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.885 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.885 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.886 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181204+0000", 2024-04-26T01:08:29.886 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0012837370000000001 2024-04-26T01:08:29.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.886 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.886 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181248+0000", 2024-04-26T01:08:29.886 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.4299999999999999e-05 2024-04-26T01:08:29.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.886 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.886 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.185079+0000", 2024-04-26T01:08:29.886 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0038302190000000002 2024-04-26T01:08:29.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.185135+0000", 2024-04-26T01:08:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.6276000000000001e-05 2024-04-26T01:08:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs4 60/44 ECSubWrite(tid=50, reqid=osd.9.0:50, at_version=60'60, trim_to=0'0, roll_forward_to=59'56))", 2024-04-26T01:08:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.190277+0000", 2024-04-26T01:08:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 3.6101023579999998, 2024-04-26T01:08:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0056685950000000002, 2024-04-26T01:08:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.888 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.888 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.888 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.888 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190277+0000", 2024-04-26T01:08:29.888 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.888 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.888 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190277+0000", 2024-04-26T01:08:29.888 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.889 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.889 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190279+0000", 2024-04-26T01:08:29.889 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9300000000000002e-06 2024-04-26T01:08:29.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.889 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.889 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.889 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190330+0000", 2024-04-26T01:08:29.889 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0798999999999999e-05 2024-04-26T01:08:29.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.889 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.889 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.889 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190406+0000", 2024-04-26T01:08:29.889 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.6406e-05 2024-04-26T01:08:29.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.890 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.890 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190427+0000", 2024-04-26T01:08:29.890 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0590000000000001e-05 2024-04-26T01:08:29.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.890 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.890 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190589+0000", 2024-04-26T01:08:29.890 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000161872 2024-04-26T01:08:29.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.890 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.890 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190626+0000", 2024-04-26T01:08:29.890 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7403000000000001e-05 2024-04-26T01:08:29.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.891 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.891 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195858+0000", 2024-04-26T01:08:29.891 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0052319289999999997 2024-04-26T01:08:29.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.891 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.891 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195946+0000", 2024-04-26T01:08:29.891 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7665999999999997e-05 2024-04-26T01:08:29.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.891 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.892 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 60/30 ECSubWrite(tid=354, reqid=osd.4.0:351, at_version=0'0, trim_to=0'0, roll_forward_to=60'54))", 2024-04-26T01:08:29.892 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.190475+0000", 2024-04-26T01:08:29.892 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 3.609904147, 2024-04-26T01:08:29.892 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0054218950000000004, 2024-04-26T01:08:29.892 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.892 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.892 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.892 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.892 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.892 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190475+0000", 2024-04-26T01:08:29.892 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.893 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.893 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190475+0000", 2024-04-26T01:08:29.893 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.893 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.893 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190478+0000", 2024-04-26T01:08:29.893 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3039999999999999e-06 2024-04-26T01:08:29.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.893 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.893 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190506+0000", 2024-04-26T01:08:29.893 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7976000000000001e-05 2024-04-26T01:08:29.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.894 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.894 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.894 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190595+0000", 2024-04-26T01:08:29.894 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9883000000000007e-05 2024-04-26T01:08:29.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.894 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.894 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.894 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190641+0000", 2024-04-26T01:08:29.894 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.5663e-05 2024-04-26T01:08:29.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.894 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.894 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.894 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190747+0000", 2024-04-26T01:08:29.894 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000106339 2024-04-26T01:08:29.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.895 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.895 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.895 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190763+0000", 2024-04-26T01:08:29.895 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5832999999999999e-05 2024-04-26T01:08:29.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.895 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.895 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.895 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195754+0000", 2024-04-26T01:08:29.895 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0049904440000000001 2024-04-26T01:08:29.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.895 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.895 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195897+0000", 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014345300000000001 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.0s5 60/30 ECSubWrite(tid=71, reqid=osd.14.0:70, at_version=0'0, trim_to=0'0, roll_forward_to=60'57))", 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.214164+0000", 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 3.5862155279999999, 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059055820000000004, 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:29.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.897 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:29.897 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214164+0000", 2024-04-26T01:08:29.897 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.897 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:29.897 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214164+0000", 2024-04-26T01:08:29.897 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:29.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.897 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:29.897 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214166+0000", 2024-04-26T01:08:29.897 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1660000000000001e-06 2024-04-26T01:08:29.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.898 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:29.898 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214194+0000", 2024-04-26T01:08:29.898 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8067000000000001e-05 2024-04-26T01:08:29.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.898 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.898 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:29.898 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214283+0000", 2024-04-26T01:08:29.898 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8609000000000005e-05 2024-04-26T01:08:29.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.898 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.898 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:29.899 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214306+0000", 2024-04-26T01:08:29.899 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3087000000000001e-05 2024-04-26T01:08:29.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.899 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:29.899 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214457+0000", 2024-04-26T01:08:29.899 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015159799999999999 2024-04-26T01:08:29.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.899 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:29.899 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214491+0000", 2024-04-26T01:08:29.900 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.3166000000000001e-05 2024-04-26T01:08:29.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.900 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.900 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:29.900 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.219992+0000", 2024-04-26T01:08:29.900 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0055015009999999998 2024-04-26T01:08:29.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:29.900 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:29.900 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:29.900 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.220069+0000", 2024-04-26T01:08:29.900 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.7387999999999998e-05 2024-04-26T01:08:29.900 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.900 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.900 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.901 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:29.901 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:29.901 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:29.902 INFO:tasks.rados.rados.0.smithi164.stdout:264: finishing write tid 6 to smithi16428812-1 2024-04-26T01:08:29.902 INFO:tasks.rados.rados.0.smithi164.stdout:264: oid 1 updating version 0 to 77 2024-04-26T01:08:29.902 INFO:tasks.rados.rados.0.smithi164.stdout:264: oid 1 updating version 77 to 78 2024-04-26T01:08:29.902 INFO:tasks.rados.rados.0.smithi164.stdout:264: oid 1 version 78 is already newer than 76 2024-04-26T01:08:29.902 INFO:tasks.rados.rados.0.smithi164.stdout:264: oid 1 version 78 is already newer than 74 2024-04-26T01:08:29.902 INFO:tasks.rados.rados.0.smithi164.stdout:264: oid 1 version 78 is already newer than 75 2024-04-26T01:08:29.902 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 1 v 78 (ObjNum 121 snap 15 seq_num 121) dirty exists 2024-04-26T01:08:29.902 INFO:tasks.rados.rados.0.smithi164.stdout:264: left oid 1 (ObjNum 121 snap 15 seq_num 121) 2024-04-26T01:08:29.903 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:08:29.904 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:29.900+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:29.951 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:08:29.951 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:08:29.993 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:29.993 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:30.003 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:30.003+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:30.029 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:08:30.029 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:30.032 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:30.032 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:30.033 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:30.028+0000 7fe8f27fc640 0 -- 172.21.15.8:0/2413155896 >> [v2:172.21.15.116:6800/2158650188,v1:172.21.15.116:6801/2158650188] conn(0x7fe8f421df70 msgr2=0x7fe8f4220360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:08:30.043 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:08:30.136 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:30.135+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:30.154 INFO:tasks.rados.rados.0.smithi164.stdout:266: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-26T01:08:30.187 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-04-26T01:08:30.192 INFO:tasks.rados.rados.0.smithi164.stdout:262: done (3 left) 2024-04-26T01:08:30.192 INFO:tasks.rados.rados.0.smithi164.stdout:264: done (2 left) 2024-04-26T01:08:30.192 INFO:tasks.rados.rados.0.smithi164.stdout:266: done (1 left) 2024-04-26T01:08:30.192 INFO:tasks.rados.rados.0.smithi164.stdout:267: done (0 left) 2024-04-26T01:08:30.193 INFO:tasks.rados.rados.0.smithi164.stdout:268: read oid 21 snap -1 2024-04-26T01:08:30.193 INFO:tasks.rados.rados.0.smithi164.stdout:268: expect (ObjNum 88 snap 6 seq_num 88) 2024-04-26T01:08:30.193 INFO:tasks.rados.rados.0.smithi164.stdout:269: copy_from oid 42 from oid 36 current snap is 16 2024-04-26T01:08:30.193 INFO:tasks.rados.rados.0.smithi164.stdout:270: snap_create 2024-04-26T01:08:30.205 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:30.203+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:30.211 INFO:tasks.rados.rados.0.smithi164.stdout:269: finishing copy_from racing read to smithi16428812-42 2024-04-26T01:08:30.213 INFO:tasks.rados.rados.0.smithi164.stdout:269: finishing copy_from to smithi16428812-42 2024-04-26T01:08:30.213 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 42 v 6 (ObjNum 69 snap 4 seq_num 69) dirty exists 2024-04-26T01:08:30.218 INFO:tasks.rados.rados.0.smithi164.stdout:268: expect (ObjNum 88 snap 6 seq_num 88) 2024-04-26T01:08:30.220 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:30.220 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:30.220 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:08:30.220 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:08:30.220 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:30.235 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:08:30.268 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:30.269 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:30.315 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:30.316 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:30.324 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:30.323+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:30.407 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:30.408 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:08:30.408 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:08:30.408 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:08:30.408 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.408 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 32/30 ECSubWrite(tid=1, reqid=client.4521.0:25, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:30.408 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:30.408 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 31.090768100999998, 2024-04-26T01:08:30.408 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062762640000000001, 2024-04-26T01:08:30.409 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.409 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.409 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.409 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.409 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.409 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.409 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:30.409 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.410 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.410 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:30.410 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.410 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.410 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:30.410 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2700000000000001e-07 2024-04-26T01:08:30.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.410 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.411 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313923+0000", 2024-04-26T01:08:30.411 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.0530000000000002e-05 2024-04-26T01:08:30.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.411 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.411 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.411 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314064+0000", 2024-04-26T01:08:30.411 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000140841 2024-04-26T01:08:30.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.411 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.411 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.411 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314070+0000", 2024-04-26T01:08:30.411 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.976e-06 2024-04-26T01:08:30.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.411 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.412 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.412 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314140+0000", 2024-04-26T01:08:30.412 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9664000000000003e-05 2024-04-26T01:08:30.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.412 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.412 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.412 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314151+0000", 2024-04-26T01:08:30.412 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1251e-05 2024-04-26T01:08:30.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.412 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.412 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.412 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320065+0000", 2024-04-26T01:08:30.412 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059141990000000002 2024-04-26T01:08:30.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.412 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.413 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.413 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320109+0000", 2024-04-26T01:08:30.413 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3575999999999997e-05 2024-04-26T01:08:30.413 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.413 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.413 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.413 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.413 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 32/30 ECSubWrite(tid=3, reqid=client.4521.0:26, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:30.413 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.316902+0000", 2024-04-26T01:08:30.413 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 31.087698602, 2024-04-26T01:08:30.413 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0057492339999999998, 2024-04-26T01:08:30.413 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.413 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.413 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.414 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.414 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.414 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.414 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.316902+0000", 2024-04-26T01:08:30.414 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.414 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.414 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.414 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.316902+0000", 2024-04-26T01:08:30.414 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.414 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.414 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.415 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.316903+0000", 2024-04-26T01:08:30.415 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2309999999999999e-06 2024-04-26T01:08:30.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.415 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.415 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.316927+0000", 2024-04-26T01:08:30.415 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4091e-05 2024-04-26T01:08:30.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.415 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.415 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.316974+0000", 2024-04-26T01:08:30.415 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.6496999999999999e-05 2024-04-26T01:08:30.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.316986+0000", 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1953000000000001e-05 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.317067+0000", 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.1122e-05 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.317079+0000", 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2517000000000001e-05 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.416 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.322622+0000", 2024-04-26T01:08:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00554216 2024-04-26T01:08:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.322651+0000", 2024-04-26T01:08:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9663e-05 2024-04-26T01:08:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.ds3 33/30 ECSubWrite(tid=5, reqid=client.4521.0:35, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.325729+0000", 2024-04-26T01:08:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 31.078871441, 2024-04-26T01:08:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079681170000000003, 2024-04-26T01:08:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325729+0000", 2024-04-26T01:08:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325729+0000", 2024-04-26T01:08:30.419 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.419 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.419 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325730+0000", 2024-04-26T01:08:30.419 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.78e-07 2024-04-26T01:08:30.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.419 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.419 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325748+0000", 2024-04-26T01:08:30.419 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8127000000000001e-05 2024-04-26T01:08:30.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.419 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.419 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325789+0000", 2024-04-26T01:08:30.420 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.1100000000000003e-05 2024-04-26T01:08:30.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.420 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.420 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325800+0000", 2024-04-26T01:08:30.420 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0256e-05 2024-04-26T01:08:30.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.420 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.420 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325859+0000", 2024-04-26T01:08:30.420 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9849999999999998e-05 2024-04-26T01:08:30.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.420 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.421 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325869+0000", 2024-04-26T01:08:30.421 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6989999999999998e-06 2024-04-26T01:08:30.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.421 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.421 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.421 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.333645+0000", 2024-04-26T01:08:30.421 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0077756800000000001 2024-04-26T01:08:30.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.421 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.421 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.421 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.333697+0000", 2024-04-26T01:08:30.421 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2426999999999998e-05 2024-04-26T01:08:30.421 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.421 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.422 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.422 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.422 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 33/30 ECSubWrite(tid=4, reqid=client.4521.0:26, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:30.422 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.327582+0000", 2024-04-26T01:08:30.422 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 31.077018801000001, 2024-04-26T01:08:30.422 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0074218790000000001, 2024-04-26T01:08:30.422 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.422 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.422 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.422 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.422 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.422 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327582+0000", 2024-04-26T01:08:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327582+0000", 2024-04-26T01:08:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327583+0000", 2024-04-26T01:08:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0669999999999999e-06 2024-04-26T01:08:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327596+0000", 2024-04-26T01:08:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3135e-05 2024-04-26T01:08:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327649+0000", 2024-04-26T01:08:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2704999999999998e-05 2024-04-26T01:08:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327659+0000", 2024-04-26T01:08:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.8470000000000008e-06 2024-04-26T01:08:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.425 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327733+0000", 2024-04-26T01:08:30.425 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4114000000000002e-05 2024-04-26T01:08:30.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.425 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.425 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327746+0000", 2024-04-26T01:08:30.425 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3176999999999999e-05 2024-04-26T01:08:30.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.425 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.425 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.334969+0000", 2024-04-26T01:08:30.425 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0072232160000000002 2024-04-26T01:08:30.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.425 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.426 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.335004+0000", 2024-04-26T01:08:30.426 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4618e-05 2024-04-26T01:08:30.426 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.426 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.426 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.426 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.426 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 33/30 ECSubWrite(tid=32, reqid=client.4521.0:134, at_version=33'8, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:08:30.426 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.467026+0000", 2024-04-26T01:08:30.426 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 30.937575028000001, 2024-04-26T01:08:30.426 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0055713610000000004, 2024-04-26T01:08:30.426 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.426 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.427 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.427 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.427 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.427 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467026+0000", 2024-04-26T01:08:30.427 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.427 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.427 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467026+0000", 2024-04-26T01:08:30.427 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.427 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.427 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467026+0000", 2024-04-26T01:08:30.428 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.6600000000000005e-07 2024-04-26T01:08:30.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.428 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.428 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.428 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467097+0000", 2024-04-26T01:08:30.428 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.0048000000000004e-05 2024-04-26T01:08:30.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.428 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.428 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.428 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467143+0000", 2024-04-26T01:08:30.428 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.6139e-05 2024-04-26T01:08:30.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.428 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.428 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.428 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467155+0000", 2024-04-26T01:08:30.429 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2123000000000001e-05 2024-04-26T01:08:30.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.429 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.429 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467373+0000", 2024-04-26T01:08:30.429 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00021788499999999999 2024-04-26T01:08:30.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.429 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.429 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.467384+0000", 2024-04-26T01:08:30.429 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0856000000000001e-05 2024-04-26T01:08:30.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.429 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.429 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472554+0000", 2024-04-26T01:08:30.430 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0051708830000000002 2024-04-26T01:08:30.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.430 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.430 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472597+0000", 2024-04-26T01:08:30.430 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.2561000000000001e-05 2024-04-26T01:08:30.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.430 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 50/44 ECSubWrite(tid=213, reqid=osd.4.0:213, at_version=50'36, trim_to=0'0, roll_forward_to=47'34))", 2024-04-26T01:08:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.047619+0000", 2024-04-26T01:08:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.356981619999999, 2024-04-26T01:08:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012971391000000001, 2024-04-26T01:08:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047619+0000", 2024-04-26T01:08:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.432 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047619+0000", 2024-04-26T01:08:30.432 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.432 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.432 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.432 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047620+0000", 2024-04-26T01:08:30.432 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9999999999999997e-07 2024-04-26T01:08:30.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.432 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.432 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.432 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047663+0000", 2024-04-26T01:08:30.432 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3254000000000003e-05 2024-04-26T01:08:30.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.432 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.432 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.433 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047762+0000", 2024-04-26T01:08:30.433 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9551999999999998e-05 2024-04-26T01:08:30.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.433 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.433 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047793+0000", 2024-04-26T01:08:30.433 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0729999999999999e-05 2024-04-26T01:08:30.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.433 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.433 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.048521+0000", 2024-04-26T01:08:30.433 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000727971 2024-04-26T01:08:30.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.433 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.434 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.048552+0000", 2024-04-26T01:08:30.434 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1375999999999999e-05 2024-04-26T01:08:30.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.434 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.434 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.434 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060503+0000", 2024-04-26T01:08:30.434 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011950771000000001 2024-04-26T01:08:30.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.434 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.434 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.434 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060590+0000", 2024-04-26T01:08:30.434 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7137000000000003e-05 2024-04-26T01:08:30.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.435 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.435 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.435 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.435 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.435 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 50/44 ECSubWrite(tid=217, reqid=osd.4.0:213, at_version=0'0, trim_to=0'0, roll_forward_to=50'36))", 2024-04-26T01:08:30.435 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.063223+0000", 2024-04-26T01:08:30.435 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.341377144000001, 2024-04-26T01:08:30.435 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0071347260000000001, 2024-04-26T01:08:30.435 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.435 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.435 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.435 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.435 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.435 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.435 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063223+0000", 2024-04-26T01:08:30.436 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.436 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.436 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.436 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063223+0000", 2024-04-26T01:08:30.436 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.436 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.436 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.436 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063225+0000", 2024-04-26T01:08:30.436 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.753e-06 2024-04-26T01:08:30.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.436 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.436 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.437 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063249+0000", 2024-04-26T01:08:30.437 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3896999999999998e-05 2024-04-26T01:08:30.437 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.437 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.437 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.437 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063327+0000", 2024-04-26T01:08:30.437 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.7693000000000003e-05 2024-04-26T01:08:30.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.438 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.438 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.438 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063349+0000", 2024-04-26T01:08:30.438 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2446000000000002e-05 2024-04-26T01:08:30.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.438 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.438 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.438 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063511+0000", 2024-04-26T01:08:30.438 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016191799999999999 2024-04-26T01:08:30.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.438 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.438 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.438 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063538+0000", 2024-04-26T01:08:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7237e-05 2024-04-26T01:08:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.070265+0000", 2024-04-26T01:08:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.006726526 2024-04-26T01:08:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.070358+0000", 2024-04-26T01:08:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.3256000000000003e-05 2024-04-26T01:08:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.440 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.440 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 50/44 ECSubWrite(tid=22, reqid=osd.9.0:22, at_version=50'37, trim_to=0'0, roll_forward_to=47'35))", 2024-04-26T01:08:30.440 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.064823+0000", 2024-04-26T01:08:30.440 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.339778022999999, 2024-04-26T01:08:30.440 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0055294560000000003, 2024-04-26T01:08:30.440 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.440 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.440 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.440 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.440 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.440 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.440 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064823+0000", 2024-04-26T01:08:30.440 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.440 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.441 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.441 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.441 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064823+0000", 2024-04-26T01:08:30.441 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.441 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.441 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.441 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064825+0000", 2024-04-26T01:08:30.441 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9e-06 2024-04-26T01:08:30.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.441 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.441 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.441 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064916+0000", 2024-04-26T01:08:30.441 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.1898999999999995e-05 2024-04-26T01:08:30.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.442 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.442 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065045+0000", 2024-04-26T01:08:30.442 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000128625 2024-04-26T01:08:30.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.442 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.442 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065069+0000", 2024-04-26T01:08:30.442 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3557000000000002e-05 2024-04-26T01:08:30.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.442 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.442 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065588+0000", 2024-04-26T01:08:30.442 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00051923300000000002 2024-04-26T01:08:30.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.443 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.443 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065664+0000", 2024-04-26T01:08:30.443 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.6229000000000005e-05 2024-04-26T01:08:30.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.443 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.443 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.070264+0000", 2024-04-26T01:08:30.443 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0046001799999999997 2024-04-26T01:08:30.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.444 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.444 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.444 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.070352+0000", 2024-04-26T01:08:30.444 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7832999999999997e-05 2024-04-26T01:08:30.444 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.444 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.444 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.445 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.445 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.1s4 50/44 ECSubWrite(tid=219, reqid=osd.4.0:219, at_version=50'58, trim_to=0'0, roll_forward_to=47'56))", 2024-04-26T01:08:30.445 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.066105+0000", 2024-04-26T01:08:30.445 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.33849612, 2024-04-26T01:08:30.445 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079801060000000007, 2024-04-26T01:08:30.445 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.445 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.445 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.446 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.446 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.446 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.446 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066105+0000", 2024-04-26T01:08:30.446 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.447 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.447 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.447 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066105+0000", 2024-04-26T01:08:30.447 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.447 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.447 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.447 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.448 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066106+0000", 2024-04-26T01:08:30.448 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.846e-06 2024-04-26T01:08:30.448 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.448 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.448 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.448 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066156+0000", 2024-04-26T01:08:30.448 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9225999999999999e-05 2024-04-26T01:08:30.448 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.449 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.449 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.449 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066238+0000", 2024-04-26T01:08:30.449 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2126999999999995e-05 2024-04-26T01:08:30.449 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.449 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.449 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.450 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066260+0000", 2024-04-26T01:08:30.450 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2408999999999998e-05 2024-04-26T01:08:30.450 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.450 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.450 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.450 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066417+0000", 2024-04-26T01:08:30.450 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000157312 2024-04-26T01:08:30.450 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.451 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.451 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.451 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066443+0000", 2024-04-26T01:08:30.451 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5797000000000001e-05 2024-04-26T01:08:30.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.451 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.451 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.452 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073990+0000", 2024-04-26T01:08:30.452 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0075467670000000002 2024-04-26T01:08:30.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.452 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.452 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.452 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.074085+0000", 2024-04-26T01:08:30.453 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.4622000000000003e-05 2024-04-26T01:08:30.453 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.453 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.453 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.453 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.453 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 50/44 ECSubWrite(tid=220, reqid=client.4521.0:491, at_version=50'38, trim_to=0'0, roll_forward_to=50'36))", 2024-04-26T01:08:30.454 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.072911+0000", 2024-04-26T01:08:30.454 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.331689598000001, 2024-04-26T01:08:30.454 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0076486990000000001, 2024-04-26T01:08:30.454 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.454 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.454 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.454 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.454 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.454 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.454 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.072911+0000", 2024-04-26T01:08:30.454 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.455 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.455 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.455 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.072911+0000", 2024-04-26T01:08:30.455 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.455 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.455 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.455 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.072914+0000", 2024-04-26T01:08:30.455 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5900000000000002e-06 2024-04-26T01:08:30.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.455 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.456 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.456 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073741+0000", 2024-04-26T01:08:30.456 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00082785699999999999 2024-04-26T01:08:30.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.456 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.456 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.456 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073888+0000", 2024-04-26T01:08:30.456 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014690499999999999 2024-04-26T01:08:30.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.456 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.456 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.456 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073915+0000", 2024-04-26T01:08:30.456 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.614e-05 2024-04-26T01:08:30.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.074074+0000", 2024-04-26T01:08:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015989500000000001 2024-04-26T01:08:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.074101+0000", 2024-04-26T01:08:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6693999999999999e-05 2024-04-26T01:08:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080467+0000", 2024-04-26T01:08:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0063662689999999999 2024-04-26T01:08:30.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.458 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.458 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.458 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080560+0000", 2024-04-26T01:08:30.458 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2349000000000006e-05 2024-04-26T01:08:30.458 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.458 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.458 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.458 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.458 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 50/44 ECSubWrite(tid=23, reqid=osd.9.0:22, at_version=0'0, trim_to=0'0, roll_forward_to=50'37))", 2024-04-26T01:08:30.458 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.079931+0000", 2024-04-26T01:08:30.458 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.324670083999999, 2024-04-26T01:08:30.459 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066276529999999998, 2024-04-26T01:08:30.459 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.459 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.459 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.459 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.459 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.459 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.459 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079931+0000", 2024-04-26T01:08:30.459 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.459 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.459 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.459 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.459 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079931+0000", 2024-04-26T01:08:30.459 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079933+0000", 2024-04-26T01:08:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6759999999999999e-06 2024-04-26T01:08:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079965+0000", 2024-04-26T01:08:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1927000000000001e-05 2024-04-26T01:08:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080073+0000", 2024-04-26T01:08:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107796 2024-04-26T01:08:30.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.461 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.461 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.461 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080098+0000", 2024-04-26T01:08:30.461 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4822999999999999e-05 2024-04-26T01:08:30.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.461 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.461 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.461 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080244+0000", 2024-04-26T01:08:30.461 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014625799999999999 2024-04-26T01:08:30.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.461 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.461 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.461 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080268+0000", 2024-04-26T01:08:30.461 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4128e-05 2024-04-26T01:08:30.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.462 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.462 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.462 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.086463+0000", 2024-04-26T01:08:30.462 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061951300000000001 2024-04-26T01:08:30.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.462 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.462 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.462 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.086558+0000", 2024-04-26T01:08:30.462 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.4914999999999995e-05 2024-04-26T01:08:30.462 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.462 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.462 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.463 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.463 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 50/44 ECSubWrite(tid=222, reqid=client.4521.0:492, at_version=50'39, trim_to=0'0, roll_forward_to=50'36))", 2024-04-26T01:08:30.463 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.080404+0000", 2024-04-26T01:08:30.463 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.324196410000001, 2024-04-26T01:08:30.463 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062500250000000002, 2024-04-26T01:08:30.463 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.463 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.463 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.463 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.463 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.463 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.463 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080404+0000", 2024-04-26T01:08:30.464 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.464 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.464 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.464 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080404+0000", 2024-04-26T01:08:30.464 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.464 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.464 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.465 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080406+0000", 2024-04-26T01:08:30.465 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0109999999999999e-06 2024-04-26T01:08:30.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.465 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.465 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.465 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080566+0000", 2024-04-26T01:08:30.465 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015976200000000001 2024-04-26T01:08:30.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.465 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.466 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.466 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080666+0000", 2024-04-26T01:08:30.466 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9698999999999995e-05 2024-04-26T01:08:30.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.466 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.466 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.466 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080693+0000", 2024-04-26T01:08:30.466 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6934e-05 2024-04-26T01:08:30.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.467 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.467 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.467 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080851+0000", 2024-04-26T01:08:30.467 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015818599999999999 2024-04-26T01:08:30.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.467 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.467 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.467 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080875+0000", 2024-04-26T01:08:30.467 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4032999999999998e-05 2024-04-26T01:08:30.468 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.468 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.468 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.468 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.086584+0000", 2024-04-26T01:08:30.468 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0057096109999999999 2024-04-26T01:08:30.468 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.468 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.469 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.469 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.086654+0000", 2024-04-26T01:08:30.469 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9789000000000006e-05 2024-04-26T01:08:30.469 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.469 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.469 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.469 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.469 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.469 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 53/44 ECSubWrite(tid=249, reqid=client.4521.0:539, at_version=53'46, trim_to=0'0, roll_forward_to=51'44))", 2024-04-26T01:08:30.470 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:19.133290+0000", 2024-04-26T01:08:30.470 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.271310566, 2024-04-26T01:08:30.470 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079087370000000008, 2024-04-26T01:08:30.470 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.470 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.470 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.470 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.470 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.470 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.471 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.133290+0000", 2024-04-26T01:08:30.471 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.471 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.471 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.471 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.471 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.133290+0000", 2024-04-26T01:08:30.471 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.471 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.472 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.472 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.472 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.133291+0000", 2024-04-26T01:08:30.472 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.4700000000000001e-07 2024-04-26T01:08:30.472 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.472 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.472 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.472 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.134718+0000", 2024-04-26T01:08:30.472 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001426921 2024-04-26T01:08:30.473 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.473 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.473 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.473 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.134818+0000", 2024-04-26T01:08:30.473 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000100203 2024-04-26T01:08:30.473 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.473 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.473 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.473 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.134841+0000", 2024-04-26T01:08:30.474 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2730000000000001e-05 2024-04-26T01:08:30.474 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.474 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.474 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.474 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.135022+0000", 2024-04-26T01:08:30.474 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00018121500000000001 2024-04-26T01:08:30.474 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.474 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.475 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.475 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.135051+0000", 2024-04-26T01:08:30.475 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8920000000000001e-05 2024-04-26T01:08:30.475 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.475 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.475 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.475 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.141103+0000", 2024-04-26T01:08:30.475 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0060520549999999998 2024-04-26T01:08:30.475 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.476 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.476 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.476 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.141199+0000", 2024-04-26T01:08:30.476 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6045999999999995e-05 2024-04-26T01:08:30.476 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.476 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.476 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.476 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.476 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.477 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 59/44 ECSubWrite(tid=323, reqid=osd.4.0:323, at_version=59'77, trim_to=0'0, roll_forward_to=58'73))", 2024-04-26T01:08:30.477 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.160765+0000", 2024-04-26T01:08:30.477 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 5.2438355469999998, 2024-04-26T01:08:30.477 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066099469999999997, 2024-04-26T01:08:30.477 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.477 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.477 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.477 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.477 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.478 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.478 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160765+0000", 2024-04-26T01:08:30.478 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.478 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.478 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.478 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.478 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160765+0000", 2024-04-26T01:08:30.478 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.479 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.479 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.479 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.479 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160766+0000", 2024-04-26T01:08:30.479 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.1999999999999999e-07 2024-04-26T01:08:30.479 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.479 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.479 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.479 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160804+0000", 2024-04-26T01:08:30.479 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8068999999999998e-05 2024-04-26T01:08:30.479 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.479 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.479 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.479 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161001+0000", 2024-04-26T01:08:30.480 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000197159 2024-04-26T01:08:30.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.480 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.480 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.480 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161025+0000", 2024-04-26T01:08:30.480 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3768999999999999e-05 2024-04-26T01:08:30.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.480 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.480 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.480 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.162206+0000", 2024-04-26T01:08:30.480 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001181398 2024-04-26T01:08:30.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.480 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.480 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.480 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.162230+0000", 2024-04-26T01:08:30.481 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.419e-05 2024-04-26T01:08:30.481 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.481 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.481 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.481 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167200+0000", 2024-04-26T01:08:30.481 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0049697099999999996 2024-04-26T01:08:30.481 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.481 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.481 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.481 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167375+0000", 2024-04-26T01:08:30.481 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017493200000000001 2024-04-26T01:08:30.481 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.481 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.481 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.481 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.482 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.482 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 59/44 ECSubWrite(tid=36, reqid=osd.9.0:36, at_version=59'49, trim_to=0'0, roll_forward_to=57'47))", 2024-04-26T01:08:30.482 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.172944+0000", 2024-04-26T01:08:30.482 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 5.2316564650000004, 2024-04-26T01:08:30.482 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0060582149999999996, 2024-04-26T01:08:30.482 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.482 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.482 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.482 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.482 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.482 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.482 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.172944+0000", 2024-04-26T01:08:30.482 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.482 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.172944+0000", 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.172945+0000", 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5000000000000002e-07 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.173004+0000", 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9209999999999997e-05 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.483 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.484 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.173118+0000", 2024-04-26T01:08:30.484 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000114422 2024-04-26T01:08:30.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.484 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.484 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.484 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.173143+0000", 2024-04-26T01:08:30.484 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4193999999999999e-05 2024-04-26T01:08:30.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.484 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.484 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.484 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.173323+0000", 2024-04-26T01:08:30.484 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000180104 2024-04-26T01:08:30.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.484 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.484 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.485 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.173340+0000", 2024-04-26T01:08:30.485 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.719e-05 2024-04-26T01:08:30.485 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.485 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.485 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.485 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.178897+0000", 2024-04-26T01:08:30.485 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0055569130000000001 2024-04-26T01:08:30.485 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.485 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.485 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.485 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.179002+0000", 2024-04-26T01:08:30.485 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010553200000000001 2024-04-26T01:08:30.486 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.486 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.486 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.486 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.486 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.486 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 59/44 ECSubWrite(tid=325, reqid=osd.4.0:323, at_version=0'0, trim_to=0'0, roll_forward_to=59'77))", 2024-04-26T01:08:30.486 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.175746+0000", 2024-04-26T01:08:30.486 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 5.2288545940000004, 2024-04-26T01:08:30.486 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0064646620000000004, 2024-04-26T01:08:30.486 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.486 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.486 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.486 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.486 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.486 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.487 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175746+0000", 2024-04-26T01:08:30.487 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.487 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.487 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.487 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175746+0000", 2024-04-26T01:08:30.487 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.487 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.487 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.487 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175748+0000", 2024-04-26T01:08:30.487 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0140000000000001e-06 2024-04-26T01:08:30.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.487 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.487 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175783+0000", 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4946000000000001e-05 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175872+0000", 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8722999999999997e-05 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175895+0000", 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3649999999999999e-05 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.488 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.176058+0000", 2024-04-26T01:08:30.489 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016219499999999999 2024-04-26T01:08:30.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.489 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.489 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.489 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.176085+0000", 2024-04-26T01:08:30.489 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7566e-05 2024-04-26T01:08:30.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.489 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.489 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.489 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.182118+0000", 2024-04-26T01:08:30.489 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0060333349999999999 2024-04-26T01:08:30.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.489 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.489 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.489 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.182211+0000", 2024-04-26T01:08:30.490 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2232999999999996e-05 2024-04-26T01:08:30.490 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.490 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.490 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.490 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.490 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/44 ECSubWrite(tid=353, reqid=osd.4.0:353, at_version=60'83, trim_to=0'0, roll_forward_to=59'79))", 2024-04-26T01:08:30.490 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.186337+0000", 2024-04-26T01:08:30.490 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.2182633369999998, 2024-04-26T01:08:30.490 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0065831279999999997, 2024-04-26T01:08:30.490 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.490 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.490 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.490 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.491 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.491 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.491 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186337+0000", 2024-04-26T01:08:30.491 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.491 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.491 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.491 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186337+0000", 2024-04-26T01:08:30.491 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.491 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.491 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.491 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186339+0000", 2024-04-26T01:08:30.492 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.08e-06 2024-04-26T01:08:30.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.492 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.492 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.492 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186388+0000", 2024-04-26T01:08:30.492 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9116000000000002e-05 2024-04-26T01:08:30.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.492 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.492 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.492 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186438+0000", 2024-04-26T01:08:30.492 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9586000000000003e-05 2024-04-26T01:08:30.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.492 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.492 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.493 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186463+0000", 2024-04-26T01:08:30.493 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4859999999999999e-05 2024-04-26T01:08:30.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.493 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.493 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186608+0000", 2024-04-26T01:08:30.493 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000145162 2024-04-26T01:08:30.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.493 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.493 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186632+0000", 2024-04-26T01:08:30.493 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4179999999999999e-05 2024-04-26T01:08:30.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.494 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.494 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.192817+0000", 2024-04-26T01:08:30.494 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061849080000000002 2024-04-26T01:08:30.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.494 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.494 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.494 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.192920+0000", 2024-04-26T01:08:30.494 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000103236 2024-04-26T01:08:30.494 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.494 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.494 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.494 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.495 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 60/44 ECSubWrite(tid=49, reqid=osd.9.0:49, at_version=60'58, trim_to=0'0, roll_forward_to=59'56))", 2024-04-26T01:08:30.495 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.187846+0000", 2024-04-26T01:08:30.495 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.2167549629999996, 2024-04-26T01:08:30.495 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0058844539999999999, 2024-04-26T01:08:30.495 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.495 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.495 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.495 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.495 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.495 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.495 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187846+0000", 2024-04-26T01:08:30.495 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.495 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.495 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.495 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.496 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187846+0000", 2024-04-26T01:08:30.496 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.496 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.496 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.496 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187846+0000", 2024-04-26T01:08:30.496 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7000000000000004e-07 2024-04-26T01:08:30.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.496 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.496 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.496 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187906+0000", 2024-04-26T01:08:30.496 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9246999999999997e-05 2024-04-26T01:08:30.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.496 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.496 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187998+0000", 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2491999999999995e-05 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.188021+0000", 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2484e-05 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.188184+0000", 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016372500000000001 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.188212+0000", 2024-04-26T01:08:30.497 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7963e-05 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.193623+0000", 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0054108280000000003 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.193730+0000", 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107045 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/44 ECSubWrite(tid=357, reqid=osd.4.0:353, at_version=0'0, trim_to=0'0, roll_forward_to=60'83))", 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.203863+0000", 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.200737803, 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0057302940000000004, 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.203863+0000", 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.203863+0000", 2024-04-26T01:08:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.203865+0000", 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6170000000000001e-06 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.203902+0000", 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6600000000000002e-05 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.203990+0000", 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7459000000000004e-05 2024-04-26T01:08:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.204011+0000", 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1895999999999998e-05 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.204205+0000", 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019383900000000001 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.204232+0000", 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6919000000000001e-05 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.502 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.209504+0000", 2024-04-26T01:08:30.502 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0052720650000000003 2024-04-26T01:08:30.502 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.502 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.502 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.502 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.209593+0000", 2024-04-26T01:08:30.502 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8899000000000004e-05 2024-04-26T01:08:30.502 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.502 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.502 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.502 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.502 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.502 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 60/44 ECSubWrite(tid=52, reqid=osd.9.0:52, at_version=60'62, trim_to=0'0, roll_forward_to=60'60))", 2024-04-26T01:08:30.502 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.205252+0000", 2024-04-26T01:08:30.503 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.199348734, 2024-04-26T01:08:30.503 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.006970324, 2024-04-26T01:08:30.503 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:30.503 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:30.503 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:30.503 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:30.503 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.503 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:30.503 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205252+0000", 2024-04-26T01:08:30.503 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.503 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.503 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.503 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:30.503 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205252+0000", 2024-04-26T01:08:30.503 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205254+0000", 2024-04-26T01:08:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8169999999999999e-06 2024-04-26T01:08:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205305+0000", 2024-04-26T01:08:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.1619000000000002e-05 2024-04-26T01:08:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205912+0000", 2024-04-26T01:08:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00060660599999999999 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205940+0000", 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7596000000000001e-05 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.206168+0000", 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00022865200000000001 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.206205+0000", 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6439000000000001e-05 2024-04-26T01:08:30.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.212015+0000", 2024-04-26T01:08:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0058099400000000004 2024-04-26T01:08:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.212222+0000", 2024-04-26T01:08:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000207655 2024-04-26T01:08:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:30.507 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:30.507 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:30.404+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:30.508 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:08:30.509 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:30.508+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:30.547 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15] out_osds: [11] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 15, 11] 2024-04-26T01:08:30.547 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-26T01:08:30.609 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:30.609+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:30.679 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:30.680 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:30.680 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:08:30.680 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:30.695 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:08:30.710 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:30.705+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:30.811 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:30.807+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:30.815 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:30.815 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:30.815 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:08:30.815 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:08:30.815 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:30.829 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:08:30.912 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:30.909+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:30.999 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:31.000 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:08:31.000 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:08:31.000 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:08:31.000 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.000 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:103 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~573440,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:31.000 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.424398+0000", 2024-04-26T01:08:31.000 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 31.573103101000001, 2024-04-26T01:08:31.000 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012920365, 2024-04-26T01:08:31.000 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.000 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.000 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:31.000 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 103 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.424398+0000", 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.424398+0000", 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.002 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.002 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.424401+0000", 2024-04-26T01:08:31.002 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0680000000000001e-06 2024-04-26T01:08:31.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.002 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.002 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.002 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425367+0000", 2024-04-26T01:08:31.002 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00096614400000000005 2024-04-26T01:08:31.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.002 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.002 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.002 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425393+0000", 2024-04-26T01:08:31.002 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.55e-05 2024-04-26T01:08:31.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.003 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.003 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425415+0000", 2024-04-26T01:08:31.003 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1959000000000001e-05 2024-04-26T01:08:31.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.003 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.003 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425510+0000", 2024-04-26T01:08:31.003 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.5134000000000001e-05 2024-04-26T01:08:31.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.003 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:31.003 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425620+0000", 2024-04-26T01:08:31.003 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00011056 2024-04-26T01:08:31.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.004 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.004 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.004 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.426564+0000", 2024-04-26T01:08:31.004 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00094343200000000002 2024-04-26T01:08:31.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.004 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.004 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.004 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428445+0000", 2024-04-26T01:08:31.004 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0018816569999999999 2024-04-26T01:08:31.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.004 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.004 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:31.004 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437169+0000", 2024-04-26T01:08:31.004 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0087233220000000004 2024-04-26T01:08:31.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.005 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.005 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437318+0000", 2024-04-26T01:08:31.005 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014958899999999999 2024-04-26T01:08:31.005 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.005 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.005 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.005 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:105 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:31.005 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:08:31.005 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 31.571980862, 2024-04-26T01:08:31.005 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017139178000000001, 2024-04-26T01:08:31.006 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.006 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.006 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:31.006 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:31.006 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:31.006 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:31.006 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 105 2024-04-26T01:08:31.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.006 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.006 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.006 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.006 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:08:31.006 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.006 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.007 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.007 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:08:31.007 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.007 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.007 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425522+0000", 2024-04-26T01:08:31.007 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5430000000000001e-06 2024-04-26T01:08:31.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.007 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.007 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425529+0000", 2024-04-26T01:08:31.007 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2570000000000001e-06 2024-04-26T01:08:31.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.008 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.008 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.008 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425548+0000", 2024-04-26T01:08:31.008 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8674000000000001e-05 2024-04-26T01:08:31.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.008 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.008 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.008 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425560+0000", 2024-04-26T01:08:31.008 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2683e-05 2024-04-26T01:08:31.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.008 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.008 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.008 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428152+0000", 2024-04-26T01:08:31.008 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0025912800000000001 2024-04-26T01:08:31.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428234+0000", 2024-04-26T01:08:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2105000000000001e-05 2024-04-26T01:08:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438095+0000", 2024-04-26T01:08:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0098614180000000003 2024-04-26T01:08:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438205+0000", 2024-04-26T01:08:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000109849 2024-04-26T01:08:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.010 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.010 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.010 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442659+0000", 2024-04-26T01:08:31.010 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0044543689999999997 2024-04-26T01:08:31.010 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.010 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.010 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.010 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.010 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:115 3.0s0 3:00e538ad:::smithi16428812-39:head [append 0~638976,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:31.010 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.437822+0000", 2024-04-26T01:08:31.010 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 31.559678526999999, 2024-04-26T01:08:31.010 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.013307608, 2024-04-26T01:08:31.010 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.011 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.011 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:31.011 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:31.011 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:31.011 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:31.011 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 115 2024-04-26T01:08:31.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.011 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.011 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.011 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437822+0000", 2024-04-26T01:08:31.011 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.011 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.012 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437822+0000", 2024-04-26T01:08:31.012 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.012 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.012 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.012 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.437824+0000", 2024-04-26T01:08:31.012 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.897e-06 2024-04-26T01:08:31.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.012 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.012 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.012 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438562+0000", 2024-04-26T01:08:31.012 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00073750400000000005 2024-04-26T01:08:31.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.012 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.013 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.013 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438572+0000", 2024-04-26T01:08:31.013 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0570999999999999e-05 2024-04-26T01:08:31.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.013 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.013 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.013 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438580+0000", 2024-04-26T01:08:31.013 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.8809999999999996e-06 2024-04-26T01:08:31.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.013 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.013 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.013 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.439078+0000", 2024-04-26T01:08:31.013 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00049728399999999996 2024-04-26T01:08:31.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.014 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.014 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:31.014 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.439233+0000", 2024-04-26T01:08:31.014 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000155134 2024-04-26T01:08:31.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.014 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.014 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.014 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.440207+0000", 2024-04-26T01:08:31.014 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00097409999999999999 2024-04-26T01:08:31.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.014 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.014 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.014 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442108+0000", 2024-04-26T01:08:31.014 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0019014139999999999 2024-04-26T01:08:31.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.015 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:31.015 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451061+0000", 2024-04-26T01:08:31.015 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0089528560000000004 2024-04-26T01:08:31.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.015 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.015 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451130+0000", 2024-04-26T01:08:31.015 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.8967000000000006e-05 2024-04-26T01:08:31.015 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.015 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.015 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:117 3.0s0 3:00e538ad:::smithi16428812-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:08:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 31.558877866, 2024-04-26T01:08:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017480893000000001, 2024-04-26T01:08:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 117 2024-04-26T01:08:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:08:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:08:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438624+0000", 2024-04-26T01:08:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.6000000000000003e-07 2024-04-26T01:08:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438628+0000", 2024-04-26T01:08:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8029999999999998e-06 2024-04-26T01:08:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438635+0000", 2024-04-26T01:08:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2799999999999998e-06 2024-04-26T01:08:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438641+0000", 2024-04-26T01:08:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5459999999999997e-06 2024-04-26T01:08:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442203+0000", 2024-04-26T01:08:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003561594 2024-04-26T01:08:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442287+0000", 2024-04-26T01:08:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.4443000000000004e-05 2024-04-26T01:08:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451430+0000", 2024-04-26T01:08:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0091427130000000002 2024-04-26T01:08:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451500+0000", 2024-04-26T01:08:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9342000000000001e-05 2024-04-26T01:08:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.456104+0000", 2024-04-26T01:08:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0046045119999999998 2024-04-26T01:08:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.021 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.021 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:152 3.0s0 3:0230bbd3:::smithi16428812-50:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:31.021 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.489313+0000", 2024-04-26T01:08:31.021 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 31.508188368999999, 2024-04-26T01:08:31.021 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.015629423, 2024-04-26T01:08:31.021 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.021 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.021 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:31.021 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:31.021 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:31.021 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:31.021 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 152 2024-04-26T01:08:31.021 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.021 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.022 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.022 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489313+0000", 2024-04-26T01:08:31.022 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.022 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.022 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489313+0000", 2024-04-26T01:08:31.022 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.022 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.022 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489314+0000", 2024-04-26T01:08:31.022 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2399999999999996e-07 2024-04-26T01:08:31.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.023 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.023 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.023 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489318+0000", 2024-04-26T01:08:31.023 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3800000000000004e-06 2024-04-26T01:08:31.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.023 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.023 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.023 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489326+0000", 2024-04-26T01:08:31.023 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5459999999999992e-06 2024-04-26T01:08:31.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.023 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.023 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.023 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489335+0000", 2024-04-26T01:08:31.024 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9050000000000003e-06 2024-04-26T01:08:31.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.024 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.024 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.491271+0000", 2024-04-26T01:08:31.024 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0019354579999999999 2024-04-26T01:08:31.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.024 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:31.024 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.491416+0000", 2024-04-26T01:08:31.024 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000145396 2024-04-26T01:08:31.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.024 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.024 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492139+0000", 2024-04-26T01:08:31.025 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00072277400000000003 2024-04-26T01:08:31.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.025 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.025 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.025 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492918+0000", 2024-04-26T01:08:31.025 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00077903499999999997 2024-04-26T01:08:31.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.025 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.025 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:31.025 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504860+0000", 2024-04-26T01:08:31.025 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011941712 2024-04-26T01:08:31.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.025 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.025 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.026 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504942+0000", 2024-04-26T01:08:31.026 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2293000000000006e-05 2024-04-26T01:08:31.026 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.026 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.026 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.026 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.026 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:153 3.0s0 3:0230bbd3:::smithi16428812-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:08:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 31.508154703999999, 2024-04-26T01:08:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017766607, 2024-04-26T01:08:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 153 2024-04-26T01:08:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:08:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:08:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489347+0000", 2024-04-26T01:08:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9999999999999997e-07 2024-04-26T01:08:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489351+0000", 2024-04-26T01:08:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7309999999999999e-06 2024-04-26T01:08:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489359+0000", 2024-04-26T01:08:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.6619999999999994e-06 2024-04-26T01:08:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489366+0000", 2024-04-26T01:08:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.798e-06 2024-04-26T01:08:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492454+0000", 2024-04-26T01:08:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0030882399999999999 2024-04-26T01:08:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492562+0000", 2024-04-26T01:08:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107948 2024-04-26T01:08:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504997+0000", 2024-04-26T01:08:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012434176 2024-04-26T01:08:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.505136+0000", 2024-04-26T01:08:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013994300000000001 2024-04-26T01:08:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.507113+0000", 2024-04-26T01:08:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0019764090000000002 2024-04-26T01:08:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.032 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.032 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.032 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.032 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:284 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~753664] snapc 4=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:31.032 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:04.530659+0000", 2024-04-26T01:08:31.032 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 26.466841555999999, 2024-04-26T01:08:31.032 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.015592622, 2024-04-26T01:08:31.032 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.032 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.032 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:31.032 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:31.032 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:31.032 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:31.033 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 284 2024-04-26T01:08:31.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.033 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.033 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.033 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.033 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.530659+0000", 2024-04-26T01:08:31.033 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.033 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.033 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.033 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.530659+0000", 2024-04-26T01:08:31.033 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.033 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.034 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.034 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.530663+0000", 2024-04-26T01:08:31.034 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7529999999999999e-06 2024-04-26T01:08:31.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.034 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.034 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.034 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.531697+0000", 2024-04-26T01:08:31.034 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0010340410000000001 2024-04-26T01:08:31.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.034 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.034 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.035 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.531730+0000", 2024-04-26T01:08:31.035 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.3194000000000001e-05 2024-04-26T01:08:31.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.035 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.035 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.035 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.531756+0000", 2024-04-26T01:08:31.035 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5559999999999999e-05 2024-04-26T01:08:31.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.035 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.035 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.035 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.531934+0000", 2024-04-26T01:08:31.035 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017789499999999999 2024-04-26T01:08:31.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.036 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.036 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:31.036 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.532110+0000", 2024-04-26T01:08:31.036 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017660500000000001 2024-04-26T01:08:31.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.036 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.036 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.036 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.534200+0000", 2024-04-26T01:08:31.036 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0020893299999999999 2024-04-26T01:08:31.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.036 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.036 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.036 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.541721+0000", 2024-04-26T01:08:31.036 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0075214430000000001 2024-04-26T01:08:31.037 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.037 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.037 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:31.037 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.546130+0000", 2024-04-26T01:08:31.037 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0044085820000000003 2024-04-26T01:08:31.037 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.037 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.037 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.037 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.546252+0000", 2024-04-26T01:08:31.037 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012221899999999999 2024-04-26T01:08:31.037 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.037 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.037 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.037 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.038 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.038 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:285 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~376832] snapc 4=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:31.038 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:04.533234+0000", 2024-04-26T01:08:31.038 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 26.464267456999998, 2024-04-26T01:08:31.038 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.013317677, 2024-04-26T01:08:31.038 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.038 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.038 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:31.038 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:31.038 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:31.038 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:31.038 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 285 2024-04-26T01:08:31.038 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.038 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.039 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.039 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.039 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533234+0000", 2024-04-26T01:08:31.039 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.039 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.039 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.039 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533234+0000", 2024-04-26T01:08:31.039 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.039 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.039 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.039 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533237+0000", 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.2030000000000001e-06 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533827+0000", 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00058986900000000003 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533847+0000", 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0789999999999999e-05 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533859+0000", 2024-04-26T01:08:31.040 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.188e-05 2024-04-26T01:08:31.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.041 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.041 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.041 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.537616+0000", 2024-04-26T01:08:31.041 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0037569159999999999 2024-04-26T01:08:31.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.041 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.041 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:31.041 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.537883+0000", 2024-04-26T01:08:31.041 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00026640499999999999 2024-04-26T01:08:31.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.042 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.042 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.042 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.539825+0000", 2024-04-26T01:08:31.042 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001942781 2024-04-26T01:08:31.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.042 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.042 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.042 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.542144+0000", 2024-04-26T01:08:31.042 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0023188850000000001 2024-04-26T01:08:31.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.042 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.042 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:31.042 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.546450+0000", 2024-04-26T01:08:31.043 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0043053019999999996 2024-04-26T01:08:31.043 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.043 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.043 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.043 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.546551+0000", 2024-04-26T01:08:31.043 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000101646 2024-04-26T01:08:31.043 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.043 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.043 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.043 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.043 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.043 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:286 3.0s0 3:0982979f:::smithi16428812-35:head [setxattr _header (56) in=7b] snapc 4=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:31.043 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:04.533890+0000", 2024-04-26T01:08:31.043 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 26.463611324999999, 2024-04-26T01:08:31.043 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014544107000000001, 2024-04-26T01:08:31.044 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.044 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.044 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:31.044 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:31.044 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:31.044 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:31.044 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 286 2024-04-26T01:08:31.044 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.044 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.044 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.044 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.044 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533890+0000", 2024-04-26T01:08:31.044 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.045 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.045 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.045 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533890+0000", 2024-04-26T01:08:31.045 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.045 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.045 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.045 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533891+0000", 2024-04-26T01:08:31.045 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.686e-06 2024-04-26T01:08:31.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.045 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.045 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.045 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533914+0000", 2024-04-26T01:08:31.046 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3087000000000001e-05 2024-04-26T01:08:31.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.046 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.046 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533934+0000", 2024-04-26T01:08:31.046 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9522999999999999e-05 2024-04-26T01:08:31.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.046 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.046 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533949+0000", 2024-04-26T01:08:31.046 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4705999999999999e-05 2024-04-26T01:08:31.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.047 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.047 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.542306+0000", 2024-04-26T01:08:31.047 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0083571340000000004 2024-04-26T01:08:31.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.047 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.047 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:31.047 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.542483+0000", 2024-04-26T01:08:31.047 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000177405 2024-04-26T01:08:31.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.047 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.047 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.047 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.543919+0000", 2024-04-26T01:08:31.047 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0014354350000000001 2024-04-26T01:08:31.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.048 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.048 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.545371+0000", 2024-04-26T01:08:31.048 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001452152 2024-04-26T01:08:31.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.048 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:31.048 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.548178+0000", 2024-04-26T01:08:31.048 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0028074380000000002 2024-04-26T01:08:31.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.048 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.049 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.548434+0000", 2024-04-26T01:08:31.049 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00025554100000000003 2024-04-26T01:08:31.049 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.049 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.049 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.049 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.049 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:287 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:31.049 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:04.533969+0000", 2024-04-26T01:08:31.049 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 26.463531632999999, 2024-04-26T01:08:31.049 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.018210147999999999, 2024-04-26T01:08:31.049 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.049 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.049 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:31.050 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:31.050 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:31.050 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:31.050 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 287 2024-04-26T01:08:31.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.050 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.050 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.050 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533969+0000", 2024-04-26T01:08:31.050 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.050 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.051 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533969+0000", 2024-04-26T01:08:31.054 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.054 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.054 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.054 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533971+0000", 2024-04-26T01:08:31.054 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3629999999999999e-06 2024-04-26T01:08:31.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.054 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.054 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.055 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533977+0000", 2024-04-26T01:08:31.055 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5799999999999997e-06 2024-04-26T01:08:31.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.055 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.055 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.534001+0000", 2024-04-26T01:08:31.055 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4006999999999999e-05 2024-04-26T01:08:31.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.055 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.055 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.534016+0000", 2024-04-26T01:08:31.055 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5160000000000001e-05 2024-04-26T01:08:31.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.055 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.056 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.545071+0000", 2024-04-26T01:08:31.056 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011054938 2024-04-26T01:08:31.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.056 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:31.056 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.545349+0000", 2024-04-26T01:08:31.056 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00027734100000000001 2024-04-26T01:08:31.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.056 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.056 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.549341+0000", 2024-04-26T01:08:31.056 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0039925150000000003 2024-04-26T01:08:31.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.057 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:31.057 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.549639+0000", 2024-04-26T01:08:31.057 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00029756800000000001 2024-04-26T01:08:31.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.057 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.057 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.057 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.552180+0000", 2024-04-26T01:08:31.057 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.002540676 2024-04-26T01:08:31.057 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.057 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.057 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.057 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.058 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:371 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:31.058 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.438556+0000", 2024-04-26T01:08:31.058 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 21.558945061999999, 2024-04-26T01:08:31.058 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.015156252, 2024-04-26T01:08:31.058 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.058 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.058 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:31.058 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:31.058 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:31.058 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:31.058 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 371 2024-04-26T01:08:31.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.059 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.059 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.059 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438556+0000", 2024-04-26T01:08:31.059 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.059 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.059 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438556+0000", 2024-04-26T01:08:31.059 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.059 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.059 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438557+0000", 2024-04-26T01:08:31.060 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4699999999999999e-06 2024-04-26T01:08:31.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.060 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.060 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438563+0000", 2024-04-26T01:08:31.060 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7699999999999998e-06 2024-04-26T01:08:31.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.060 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.060 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438608+0000", 2024-04-26T01:08:31.060 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.4898999999999998e-05 2024-04-26T01:08:31.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.060 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.060 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438623+0000", 2024-04-26T01:08:31.061 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.47e-05 2024-04-26T01:08:31.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.061 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.061 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.450782+0000", 2024-04-26T01:08:31.061 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012158983 2024-04-26T01:08:31.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.061 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:31.061 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.450982+0000", 2024-04-26T01:08:31.061 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000200602 2024-04-26T01:08:31.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.061 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.062 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.453712+0000", 2024-04-26T01:08:31.062 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0027298280000000001 2024-04-26T01:08:31.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.062 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.062 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:542 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~606208] snapc e=[e,d,c,b,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-04-26T01:08:31.062 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:20.149139+0000", 2024-04-26T01:08:31.062 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.848362058999999, 2024-04-26T01:08:31.062 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016978235000000001, 2024-04-26T01:08:31.062 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.063 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.063 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:31.063 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:31.063 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:31.063 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:31.063 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 542 2024-04-26T01:08:31.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.063 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.064 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.064 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.149139+0000", 2024-04-26T01:08:31.064 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.064 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.064 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.064 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.149139+0000", 2024-04-26T01:08:31.065 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.065 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.065 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.065 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.149142+0000", 2024-04-26T01:08:31.065 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9639999999999999e-06 2024-04-26T01:08:31.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.065 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.065 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.065 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150216+0000", 2024-04-26T01:08:31.065 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001074478 2024-04-26T01:08:31.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.066 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.066 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.066 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150251+0000", 2024-04-26T01:08:31.066 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4563000000000001e-05 2024-04-26T01:08:31.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.066 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.066 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.066 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150269+0000", 2024-04-26T01:08:31.066 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.806e-05 2024-04-26T01:08:31.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.066 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.066 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.066 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150398+0000", 2024-04-26T01:08:31.066 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012864500000000001 2024-04-26T01:08:31.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.067 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.067 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "op must wait for map", 2024-04-26T01:08:31.067 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150410+0000", 2024-04-26T01:08:31.067 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2427e-05 2024-04-26T01:08:31.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.067 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.067 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.067 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.154650+0000", 2024-04-26T01:08:31.067 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0042403129999999999 2024-04-26T01:08:31.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.067 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.068 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:31.068 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.154806+0000", 2024-04-26T01:08:31.068 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015539199999999999 2024-04-26T01:08:31.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.068 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.068 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.068 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.157843+0000", 2024-04-26T01:08:31.068 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0030374519999999999 2024-04-26T01:08:31.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.068 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.068 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.068 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.165692+0000", 2024-04-26T01:08:31.068 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0078487009999999996 2024-04-26T01:08:31.069 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.069 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.069 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:31.069 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.165984+0000", 2024-04-26T01:08:31.069 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00029256099999999999 2024-04-26T01:08:31.069 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.069 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.069 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.069 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.166117+0000", 2024-04-26T01:08:31.069 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013267899999999999 2024-04-26T01:08:31.069 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.069 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.070 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.070 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.070 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:543 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~688128] snapc e=[e,d,c,b,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-04-26T01:08:31.070 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:20.158153+0000", 2024-04-26T01:08:31.070 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.839348394, 2024-04-26T01:08:31.070 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014883451000000001, 2024-04-26T01:08:31.070 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.070 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.070 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:31.070 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:31.070 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:31.070 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:31.070 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 543 2024-04-26T01:08:31.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.071 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.071 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.071 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.071 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.158153+0000", 2024-04-26T01:08:31.071 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.071 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.071 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.071 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.158153+0000", 2024-04-26T01:08:31.071 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.071 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.071 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.072 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.158156+0000", 2024-04-26T01:08:31.072 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.2969999999999999e-06 2024-04-26T01:08:31.072 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.072 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.072 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.072 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.159404+0000", 2024-04-26T01:08:31.072 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001248426 2024-04-26T01:08:31.072 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.072 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.072 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.072 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.159440+0000", 2024-04-26T01:08:31.072 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5284000000000003e-05 2024-04-26T01:08:31.072 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.072 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.072 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.073 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.159462+0000", 2024-04-26T01:08:31.073 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2682999999999999e-05 2024-04-26T01:08:31.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.073 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.073 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.073 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.160794+0000", 2024-04-26T01:08:31.073 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001331336 2024-04-26T01:08:31.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.073 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.073 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:31.073 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.161097+0000", 2024-04-26T01:08:31.073 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00030376699999999998 2024-04-26T01:08:31.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.074 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.074 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.074 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.163627+0000", 2024-04-26T01:08:31.074 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0025294900000000001 2024-04-26T01:08:31.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.074 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.074 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.074 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.172181+0000", 2024-04-26T01:08:31.074 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0085543579999999998 2024-04-26T01:08:31.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.074 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.074 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:31.075 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.172841+0000", 2024-04-26T01:08:31.075 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00065971499999999995 2024-04-26T01:08:31.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.075 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.075 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.075 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.173036+0000", 2024-04-26T01:08:31.075 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000195095 2024-04-26T01:08:31.075 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.075 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.075 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.076 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.4s5 55/30 ECSubWrite(tid=39, reqid=osd.8.0:39, at_version=55'26, trim_to=0'0, roll_forward_to=52'24))", 2024-04-26T01:08:31.076 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:21.091694+0000", 2024-04-26T01:08:31.076 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 9.9058067760000004, 2024-04-26T01:08:31.076 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.49007189200000001, 2024-04-26T01:08:31.076 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.076 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.076 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.076 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.076 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.076 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091694+0000", 2024-04-26T01:08:31.076 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091694+0000", 2024-04-26T01:08:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091695+0000", 2024-04-26T01:08:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.0699999999999996e-07 2024-04-26T01:08:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091753+0000", 2024-04-26T01:08:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7516999999999998e-05 2024-04-26T01:08:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580278+0000", 2024-04-26T01:08:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.48852570099999998 2024-04-26T01:08:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580292+0000", 2024-04-26T01:08:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3671e-05 2024-04-26T01:08:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580396+0000", 2024-04-26T01:08:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010389400000000001 2024-04-26T01:08:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580413+0000", 2024-04-26T01:08:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7121000000000001e-05 2024-04-26T01:08:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.581731+0000", 2024-04-26T01:08:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001318294 2024-04-26T01:08:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.080 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.080 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.080 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.581766+0000", 2024-04-26T01:08:31.080 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4786999999999997e-05 2024-04-26T01:08:31.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.080 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.080 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.080 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 58/30 ECSubWriteReply(tid=65, last_complete=58'49, committed=1, applied=1))", 2024-04-26T01:08:31.080 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:24.344800+0000", 2024-04-26T01:08:31.080 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 6.6527010689999999, 2024-04-26T01:08:31.080 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.33836068000000002, 2024-04-26T01:08:31.080 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.081 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.081 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.081 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.081 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.081 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344800+0000", 2024-04-26T01:08:31.081 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.081 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.081 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344800+0000", 2024-04-26T01:08:31.081 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.082 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.082 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344802+0000", 2024-04-26T01:08:31.082 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.593e-06 2024-04-26T01:08:31.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.082 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.082 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.082 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344812+0000", 2024-04-26T01:08:31.082 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0943e-05 2024-04-26T01:08:31.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.082 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.082 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.082 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.682937+0000", 2024-04-26T01:08:31.083 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.33812443599999997 2024-04-26T01:08:31.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.083 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.083 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.682952+0000", 2024-04-26T01:08:31.083 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5116000000000001e-05 2024-04-26T01:08:31.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.083 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.083 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.683088+0000", 2024-04-26T01:08:31.083 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000135969 2024-04-26T01:08:31.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.083 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.084 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.683161+0000", 2024-04-26T01:08:31.084 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2622999999999999e-05 2024-04-26T01:08:31.084 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.084 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.084 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.084 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.4s5 60/30 ECSubWrite(tid=57, reqid=osd.8.0:57, at_version=60'42, trim_to=0'0, roll_forward_to=59'40))", 2024-04-26T01:08:31.084 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.181768+0000", 2024-04-26T01:08:31.084 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.8157327509999996, 2024-04-26T01:08:31.084 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.013955675000000001, 2024-04-26T01:08:31.084 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.084 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.084 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.084 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.085 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.085 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181768+0000", 2024-04-26T01:08:31.085 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.085 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.085 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181768+0000", 2024-04-26T01:08:31.085 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.085 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.085 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181769+0000", 2024-04-26T01:08:31.085 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.6300000000000005e-07 2024-04-26T01:08:31.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.086 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.086 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181827+0000", 2024-04-26T01:08:31.086 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7899000000000002e-05 2024-04-26T01:08:31.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.086 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.086 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.182172+0000", 2024-04-26T01:08:31.086 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00034487100000000001 2024-04-26T01:08:31.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.086 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.087 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.182201+0000", 2024-04-26T01:08:31.087 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9753000000000001e-05 2024-04-26T01:08:31.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.087 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.087 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.182360+0000", 2024-04-26T01:08:31.087 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000158565 2024-04-26T01:08:31.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.087 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.087 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.182387+0000", 2024-04-26T01:08:31.087 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7226999999999999e-05 2024-04-26T01:08:31.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.088 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.088 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195645+0000", 2024-04-26T01:08:31.088 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.013258135000000001 2024-04-26T01:08:31.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.088 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.088 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195724+0000", 2024-04-26T01:08:31.088 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.8662e-05 2024-04-26T01:08:31.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.088 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.088 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.4s5 60/30 ECSubWrite(tid=58, reqid=osd.8.0:58, at_version=60'44, trim_to=0'0, roll_forward_to=59'40))", 2024-04-26T01:08:31.089 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.181832+0000", 2024-04-26T01:08:31.089 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.8156688489999997, 2024-04-26T01:08:31.089 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.020381277999999999, 2024-04-26T01:08:31.089 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.089 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.089 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.089 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.089 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.089 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.089 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181832+0000", 2024-04-26T01:08:31.089 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.089 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.089 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.090 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181832+0000", 2024-04-26T01:08:31.090 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.090 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.090 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.090 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181833+0000", 2024-04-26T01:08:31.090 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.0999999999999998e-07 2024-04-26T01:08:31.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.090 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.090 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.090 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181871+0000", 2024-04-26T01:08:31.090 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8352999999999997e-05 2024-04-26T01:08:31.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.091 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.091 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.182274+0000", 2024-04-26T01:08:31.091 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00040273700000000002 2024-04-26T01:08:31.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.091 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.091 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.182291+0000", 2024-04-26T01:08:31.091 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6949999999999999e-05 2024-04-26T01:08:31.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.091 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.091 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187997+0000", 2024-04-26T01:08:31.092 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.005706198 2024-04-26T01:08:31.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.092 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.092 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.188021+0000", 2024-04-26T01:08:31.092 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3652999999999999e-05 2024-04-26T01:08:31.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.092 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.092 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.202131+0000", 2024-04-26T01:08:31.092 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014110531000000001 2024-04-26T01:08:31.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.092 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.093 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.202213+0000", 2024-04-26T01:08:31.093 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2145999999999996e-05 2024-04-26T01:08:31.093 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.093 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.093 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.093 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.093 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs2 60/44 ECSubWrite(tid=353, reqid=osd.4.0:353, at_version=60'83, trim_to=0'0, roll_forward_to=59'79))", 2024-04-26T01:08:31.093 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.186301+0000", 2024-04-26T01:08:31.093 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.8111999079999999, 2024-04-26T01:08:31.093 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016791533000000001, 2024-04-26T01:08:31.093 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.093 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.093 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.093 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.094 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.094 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.094 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186301+0000", 2024-04-26T01:08:31.094 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.094 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.094 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.094 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186301+0000", 2024-04-26T01:08:31.094 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.094 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.094 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.094 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186304+0000", 2024-04-26T01:08:31.094 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4099999999999998e-06 2024-04-26T01:08:31.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.095 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.095 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.095 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186355+0000", 2024-04-26T01:08:31.095 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.1236000000000002e-05 2024-04-26T01:08:31.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.095 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.095 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.095 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186442+0000", 2024-04-26T01:08:31.095 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7143000000000002e-05 2024-04-26T01:08:31.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.095 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.095 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.095 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186463+0000", 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0959999999999999e-05 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187730+0000", 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00126672 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187762+0000", 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.2759999999999998e-05 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.203020+0000", 2024-04-26T01:08:31.096 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.015257409 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.203093+0000", 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2894999999999999e-05 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s6 60/44 ECSubWrite(tid=355, reqid=osd.4.0:355, at_version=60'64, trim_to=0'0, roll_forward_to=59'62))", 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.194637+0000", 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.8028635130000001, 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.013455195, 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.097 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.098 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.098 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.098 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.098 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.098 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.194637+0000", 2024-04-26T01:08:31.098 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.098 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.098 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.098 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.194637+0000", 2024-04-26T01:08:31.098 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.098 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.099 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.099 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.194640+0000", 2024-04-26T01:08:31.099 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0260000000000002e-06 2024-04-26T01:08:31.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.099 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.099 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.194694+0000", 2024-04-26T01:08:31.099 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.4616e-05 2024-04-26T01:08:31.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.099 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.099 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.194958+0000", 2024-04-26T01:08:31.099 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00026364100000000001 2024-04-26T01:08:31.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.100 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.100 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195005+0000", 2024-04-26T01:08:31.100 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.7033999999999997e-05 2024-04-26T01:08:31.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.100 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.100 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195206+0000", 2024-04-26T01:08:31.100 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000201521 2024-04-26T01:08:31.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.100 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.100 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195231+0000", 2024-04-26T01:08:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4530000000000001e-05 2024-04-26T01:08:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.208012+0000", 2024-04-26T01:08:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012781054999999999 2024-04-26T01:08:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.208093+0000", 2024-04-26T01:08:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.0772000000000005e-05 2024-04-26T01:08:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.102 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s6 60/44 ECSubWrite(tid=356, reqid=osd.4.0:356, at_version=60'66, trim_to=0'0, roll_forward_to=59'62))", 2024-04-26T01:08:31.102 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.195714+0000", 2024-04-26T01:08:31.102 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.8017868220000004, 2024-04-26T01:08:31.102 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012776084, 2024-04-26T01:08:31.102 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:31.102 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:31.102 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.102 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:31.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.102 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:31.103 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195714+0000", 2024-04-26T01:08:31.103 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.103 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:31.103 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195714+0000", 2024-04-26T01:08:31.103 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:31.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.103 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:31.103 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195716+0000", 2024-04-26T01:08:31.103 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1900000000000002e-06 2024-04-26T01:08:31.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.104 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:31.104 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195764+0000", 2024-04-26T01:08:31.104 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.7605999999999998e-05 2024-04-26T01:08:31.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.104 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:31.104 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195805+0000", 2024-04-26T01:08:31.104 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.1083e-05 2024-04-26T01:08:31.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.104 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.104 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195826+0000", 2024-04-26T01:08:31.104 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1226e-05 2024-04-26T01:08:31.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.105 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:31.105 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.198111+0000", 2024-04-26T01:08:31.105 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0022849020000000001 2024-04-26T01:08:31.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.105 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.105 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.105 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.198130+0000", 2024-04-26T01:08:31.105 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8804000000000002e-05 2024-04-26T01:08:31.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.105 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.105 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.105 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.208414+0000", 2024-04-26T01:08:31.105 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010284438 2024-04-26T01:08:31.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:31.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:31.106 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:31.106 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.208490+0000", 2024-04-26T01:08:31.106 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5834999999999995e-05 2024-04-26T01:08:31.106 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.106 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.106 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.106 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:31.106 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:31.106 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:31.109 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.0 to 1 (10000) 2024-04-26T01:08:31.109 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:08:31.110 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-26T01:08:31.111 INFO:tasks.rados.rados.0.smithi164.stdout:268: done (2 left) 2024-04-26T01:08:31.111 INFO:tasks.rados.rados.0.smithi164.stdout:269: done (1 left) 2024-04-26T01:08:31.111 INFO:tasks.rados.rados.0.smithi164.stdout:270: done (0 left) 2024-04-26T01:08:31.111 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 34 current snap is 17 2024-04-26T01:08:31.111 INFO:tasks.rados.rados.0.smithi164.stdout:271: seq_num 122 ranges {0=507904,507904=425984,933888=442368,1376256=540672,1916928=147456} 2024-04-26T01:08:31.112 INFO:tasks.rados.rados.0.smithi164.stdout:271: writing smithi16428812-34 from 0 to 507904 tid 1 2024-04-26T01:08:31.112 INFO:tasks.rados.rados.0.smithi164.stdout:271: writing smithi16428812-34 from 507904 to 933888 tid 2 2024-04-26T01:08:31.112 INFO:tasks.rados.rados.0.smithi164.stdout:271: writing smithi16428812-34 from 933888 to 1376256 tid 3 2024-04-26T01:08:31.112 INFO:tasks.rados.rados.0.smithi164.stdout:271: writing smithi16428812-34 from 1376256 to 1916928 tid 4 2024-04-26T01:08:31.112 INFO:tasks.rados.rados.0.smithi164.stdout:271: writing smithi16428812-34 from 1916928 to 2064384 tid 5 2024-04-26T01:08:31.112 INFO:tasks.rados.rados.0.smithi164.stdout:272: delete oid 45 current snap is 17 2024-04-26T01:08:31.112 INFO:tasks.rados.rados.0.smithi164.stdout:271: finishing write tid 1 to smithi16428812-34 2024-04-26T01:08:31.112 INFO:tasks.rados.rados.0.smithi164.stdout:271: finishing write tid 2 to smithi16428812-34 2024-04-26T01:08:31.112 INFO:tasks.rados.rados.0.smithi164.stdout:271: finishing write tid 3 to smithi16428812-34 2024-04-26T01:08:31.112 INFO:tasks.rados.rados.0.smithi164.stdout:272: done (1 left) 2024-04-26T01:08:31.112 INFO:tasks.rados.rados.0.smithi164.stdout:273: setattr oid 30 current snap is 17 2024-04-26T01:08:31.112 INFO:tasks.rados.rados.0.smithi164.stdout:271: finishing write tid 4 to smithi16428812-34 2024-04-26T01:08:31.112 INFO:tasks.rados.rados.0.smithi164.stdout:274: snap_remove snap 16 2024-04-26T01:08:31.113 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:31.105+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:31.209 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:31.208+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:31.242 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:31.242 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:31.242 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:08:31.242 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:31.258 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:08:31.309 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:31.304+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:31.411 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:31.408+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:31.423 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:31.423 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:31.424 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2024-04-26T01:08:31.424 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2024-04-26T01:08:31.424 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:31.437 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:08:31.511 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:31.505+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:31.557 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2024-04-26T01:08:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2024-04-26T01:08:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2024-04-26T01:08:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.ds1 32/30 ECSubWrite(tid=0, reqid=client.4521.0:4, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.304352+0000", 2024-04-26T01:08:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.251687482000001, 2024-04-26T01:08:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0083134550000000008, 2024-04-26T01:08:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304352+0000", 2024-04-26T01:08:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304352+0000", 2024-04-26T01:08:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304353+0000", 2024-04-26T01:08:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5700000000000002e-07 2024-04-26T01:08:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304550+0000", 2024-04-26T01:08:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019731600000000001 2024-04-26T01:08:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304683+0000", 2024-04-26T01:08:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000132998 2024-04-26T01:08:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304726+0000", 2024-04-26T01:08:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3081999999999998e-05 2024-04-26T01:08:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304949+0000", 2024-04-26T01:08:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022243600000000001 2024-04-26T01:08:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304976+0000", 2024-04-26T01:08:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7206e-05 2024-04-26T01:08:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.312588+0000", 2024-04-26T01:08:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0076121670000000004 2024-04-26T01:08:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.312666+0000", 2024-04-26T01:08:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7593000000000001e-05 2024-04-26T01:08:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.ds1 32/30 ECSubWrite(tid=1, reqid=client.4521.0:5, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.308139+0000", 2024-04-26T01:08:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.247900706999999, 2024-04-26T01:08:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0045765040000000003, 2024-04-26T01:08:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308139+0000", 2024-04-26T01:08:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308139+0000", 2024-04-26T01:08:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308141+0000", 2024-04-26T01:08:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.778e-06 2024-04-26T01:08:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308178+0000", 2024-04-26T01:08:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6869000000000002e-05 2024-04-26T01:08:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308277+0000", 2024-04-26T01:08:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.946e-05 2024-04-26T01:08:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308297+0000", 2024-04-26T01:08:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0157e-05 2024-04-26T01:08:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308469+0000", 2024-04-26T01:08:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017191400000000001 2024-04-26T01:08:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308489+0000", 2024-04-26T01:08:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9694000000000002e-05 2024-04-26T01:08:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.312673+0000", 2024-04-26T01:08:31.568 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0041840200000000001 2024-04-26T01:08:31.568 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.568 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.568 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.568 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.312716+0000", 2024-04-26T01:08:31.568 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2611999999999998e-05 2024-04-26T01:08:31.568 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.568 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.568 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.568 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.568 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.568 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 32/30 ECSubWrite(tid=157, reqid=client.4521.0:13, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:31.569 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.310202+0000", 2024-04-26T01:08:31.569 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.245837770000001, 2024-04-26T01:08:31.569 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0083556310000000005, 2024-04-26T01:08:31.569 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.569 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.569 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.569 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.569 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.569 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.569 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.310202+0000", 2024-04-26T01:08:31.569 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.569 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.569 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.570 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.570 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.310202+0000", 2024-04-26T01:08:31.570 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.570 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.570 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.570 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.570 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.310203+0000", 2024-04-26T01:08:31.570 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.257e-06 2024-04-26T01:08:31.570 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.570 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.570 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.571 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.310319+0000", 2024-04-26T01:08:31.571 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000115784 2024-04-26T01:08:31.571 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.571 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.571 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.571 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314884+0000", 2024-04-26T01:08:31.571 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0045650969999999997 2024-04-26T01:08:31.571 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.571 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.571 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.571 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314909+0000", 2024-04-26T01:08:31.571 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4329e-05 2024-04-26T01:08:31.571 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.572 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.572 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.572 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315108+0000", 2024-04-26T01:08:31.572 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019895399999999999 2024-04-26T01:08:31.572 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.572 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.572 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.572 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315129+0000", 2024-04-26T01:08:31.572 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1662e-05 2024-04-26T01:08:31.572 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.572 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.572 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.573 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.318477+0000", 2024-04-26T01:08:31.573 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033479120000000002 2024-04-26T01:08:31.573 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.573 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.573 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.573 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.318558+0000", 2024-04-26T01:08:31.573 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.0636000000000005e-05 2024-04-26T01:08:31.573 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.573 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.573 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.573 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.573 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.573 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 32/30 ECSubWrite(tid=158, reqid=client.4521.0:14, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:31.574 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.314747+0000", 2024-04-26T01:08:31.574 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.241293300000002, 2024-04-26T01:08:31.574 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.020884725999999999, 2024-04-26T01:08:31.574 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.574 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.574 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.574 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.574 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.574 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.574 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314747+0000", 2024-04-26T01:08:31.574 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.575 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.575 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.575 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314747+0000", 2024-04-26T01:08:31.575 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.575 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.575 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.575 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314748+0000", 2024-04-26T01:08:31.575 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8700000000000001e-06 2024-04-26T01:08:31.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.575 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.575 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.576 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314821+0000", 2024-04-26T01:08:31.576 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2556000000000001e-05 2024-04-26T01:08:31.576 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.576 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.576 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.576 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323944+0000", 2024-04-26T01:08:31.576 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0091227779999999998 2024-04-26T01:08:31.576 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.576 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.576 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.576 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323963+0000", 2024-04-26T01:08:31.576 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9595000000000001e-05 2024-04-26T01:08:31.577 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.577 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.577 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.577 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324110+0000", 2024-04-26T01:08:31.577 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014636 2024-04-26T01:08:31.577 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.577 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.577 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.577 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324134+0000", 2024-04-26T01:08:31.577 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4453000000000001e-05 2024-04-26T01:08:31.577 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.577 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.578 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.578 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335532+0000", 2024-04-26T01:08:31.578 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011398118 2024-04-26T01:08:31.578 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.578 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.578 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.578 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335631+0000", 2024-04-26T01:08:31.578 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8995999999999999e-05 2024-04-26T01:08:31.578 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.578 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.578 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.578 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.579 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.579 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.ds1 33/30 ECSubWrite(tid=2, reqid=client.4521.0:5, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:31.579 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.316813+0000", 2024-04-26T01:08:31.579 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.239226936999998, 2024-04-26T01:08:31.579 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0064126929999999997, 2024-04-26T01:08:31.579 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.579 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.579 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.579 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.579 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.579 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.579 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316813+0000", 2024-04-26T01:08:31.580 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.580 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.580 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.580 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.580 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316813+0000", 2024-04-26T01:08:31.580 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.580 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.580 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.580 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.580 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316815+0000", 2024-04-26T01:08:31.580 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6649999999999999e-06 2024-04-26T01:08:31.580 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.581 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.581 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.581 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316842+0000", 2024-04-26T01:08:31.581 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7072000000000001e-05 2024-04-26T01:08:31.581 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.581 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.581 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.581 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316923+0000", 2024-04-26T01:08:31.581 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1705999999999993e-05 2024-04-26T01:08:31.581 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.581 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.582 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.582 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316942+0000", 2024-04-26T01:08:31.582 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8559e-05 2024-04-26T01:08:31.582 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.582 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.582 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.582 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.317062+0000", 2024-04-26T01:08:31.582 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011990299999999999 2024-04-26T01:08:31.582 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.582 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.582 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.582 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.317081+0000", 2024-04-26T01:08:31.583 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9312000000000001e-05 2024-04-26T01:08:31.583 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.583 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.583 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.583 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323159+0000", 2024-04-26T01:08:31.583 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0060774440000000004 2024-04-26T01:08:31.583 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.583 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.583 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.583 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323226+0000", 2024-04-26T01:08:31.583 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7032000000000005e-05 2024-04-26T01:08:31.583 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.584 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.584 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.584 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.584 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.584 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 32/30 ECSubWrite(tid=159, reqid=client.4521.0:31, at_version=32'3, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:31.584 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.323697+0000", 2024-04-26T01:08:31.584 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.232343016000002, 2024-04-26T01:08:31.584 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013382949, 2024-04-26T01:08:31.584 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.584 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.584 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.584 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.585 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.585 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.585 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323697+0000", 2024-04-26T01:08:31.585 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.585 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.585 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.585 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.585 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323697+0000", 2024-04-26T01:08:31.585 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.585 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.585 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.585 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.586 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323697+0000", 2024-04-26T01:08:31.586 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4000000000000001e-07 2024-04-26T01:08:31.586 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.586 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.586 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.586 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323896+0000", 2024-04-26T01:08:31.586 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019882299999999999 2024-04-26T01:08:31.586 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.586 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.586 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.586 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.328908+0000", 2024-04-26T01:08:31.586 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0050118239999999998 2024-04-26T01:08:31.587 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.587 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.587 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.587 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.328926+0000", 2024-04-26T01:08:31.587 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8328000000000001e-05 2024-04-26T01:08:31.587 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.587 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.587 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.587 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329545+0000", 2024-04-26T01:08:31.587 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00061852999999999995 2024-04-26T01:08:31.587 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.587 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.587 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.588 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329608+0000", 2024-04-26T01:08:31.588 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2861999999999995e-05 2024-04-26T01:08:31.588 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.588 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.588 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.588 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336992+0000", 2024-04-26T01:08:31.588 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0073843040000000004 2024-04-26T01:08:31.588 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.588 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.588 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.588 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337080+0000", 2024-04-26T01:08:31.589 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.7638000000000004e-05 2024-04-26T01:08:31.589 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.589 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.589 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.589 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.589 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.589 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.ds1 33/30 ECSubWrite(tid=4, reqid=client.4521.0:34, at_version=33'3, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:31.589 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.324243+0000", 2024-04-26T01:08:31.589 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.231796731999999, 2024-04-26T01:08:31.589 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011396646, 2024-04-26T01:08:31.589 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.589 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.590 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.590 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.590 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.590 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.590 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324243+0000", 2024-04-26T01:08:31.590 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.590 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.590 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.590 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.590 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324243+0000", 2024-04-26T01:08:31.590 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.590 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.591 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.591 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.591 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324245+0000", 2024-04-26T01:08:31.591 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2280000000000001e-06 2024-04-26T01:08:31.591 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.591 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.591 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.591 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324397+0000", 2024-04-26T01:08:31.591 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015143700000000001 2024-04-26T01:08:31.591 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.591 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.592 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.592 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324490+0000", 2024-04-26T01:08:31.592 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3133000000000004e-05 2024-04-26T01:08:31.592 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.592 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.592 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.592 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324513+0000", 2024-04-26T01:08:31.592 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2864999999999999e-05 2024-04-26T01:08:31.592 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.592 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.592 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.592 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324679+0000", 2024-04-26T01:08:31.593 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000166525 2024-04-26T01:08:31.593 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.593 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.593 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.593 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324694+0000", 2024-04-26T01:08:31.593 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4825e-05 2024-04-26T01:08:31.593 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.593 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.593 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.593 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335561+0000", 2024-04-26T01:08:31.593 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010867185999999999 2024-04-26T01:08:31.593 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.594 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.594 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.594 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335640+0000", 2024-04-26T01:08:31.594 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8447000000000003e-05 2024-04-26T01:08:31.594 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.594 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.594 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.594 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.594 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.594 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.ds1 33/30 ECSubWrite(tid=5, reqid=client.4521.0:35, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:31.594 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.326781+0000", 2024-04-26T01:08:31.595 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.229258995999999, 2024-04-26T01:08:31.595 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.008898787, 2024-04-26T01:08:31.595 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.595 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.595 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.595 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.595 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.595 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.595 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326781+0000", 2024-04-26T01:08:31.595 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.595 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.595 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.595 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.596 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326781+0000", 2024-04-26T01:08:31.596 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.596 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.596 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.596 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.596 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326783+0000", 2024-04-26T01:08:31.596 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7999999999999999e-06 2024-04-26T01:08:31.596 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.596 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.596 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.596 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326824+0000", 2024-04-26T01:08:31.597 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1421999999999997e-05 2024-04-26T01:08:31.597 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.597 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.597 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.597 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326946+0000", 2024-04-26T01:08:31.597 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012172600000000001 2024-04-26T01:08:31.597 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.597 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.597 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.597 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326968+0000", 2024-04-26T01:08:31.597 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2274000000000001e-05 2024-04-26T01:08:31.597 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.598 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.598 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.598 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.327108+0000", 2024-04-26T01:08:31.598 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000139703 2024-04-26T01:08:31.598 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.598 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.598 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.598 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.327130+0000", 2024-04-26T01:08:31.598 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2011e-05 2024-04-26T01:08:31.598 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.598 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.598 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.599 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335646+0000", 2024-04-26T01:08:31.599 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0085158969999999997 2024-04-26T01:08:31.599 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.599 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.599 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.599 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335680+0000", 2024-04-26T01:08:31.599 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3954000000000001e-05 2024-04-26T01:08:31.599 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.599 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.599 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.599 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.599 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.600 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/30 ECSubWrite(tid=160, reqid=client.4521.0:32, at_version=33'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:31.600 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.328826+0000", 2024-04-26T01:08:31.600 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.227213839000001, 2024-04-26T01:08:31.600 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010843787000000001, 2024-04-26T01:08:31.600 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.600 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.600 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.600 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.600 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.600 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.600 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.328826+0000", 2024-04-26T01:08:31.600 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.600 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.601 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.601 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.601 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.328826+0000", 2024-04-26T01:08:31.601 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.601 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.601 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.601 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.601 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.328828+0000", 2024-04-26T01:08:31.601 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7570000000000001e-06 2024-04-26T01:08:31.601 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.601 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.601 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.601 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.328866+0000", 2024-04-26T01:08:31.602 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8260000000000003e-05 2024-04-26T01:08:31.602 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.602 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.602 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.602 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337463+0000", 2024-04-26T01:08:31.602 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0085968439999999993 2024-04-26T01:08:31.602 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.602 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.602 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.602 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337477+0000", 2024-04-26T01:08:31.602 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4620999999999999e-05 2024-04-26T01:08:31.602 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.602 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.603 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.603 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337694+0000", 2024-04-26T01:08:31.603 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00021669500000000001 2024-04-26T01:08:31.603 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.603 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.603 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.603 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337715+0000", 2024-04-26T01:08:31.603 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0503e-05 2024-04-26T01:08:31.603 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.603 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.603 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.603 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339557+0000", 2024-04-26T01:08:31.603 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00184206 2024-04-26T01:08:31.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.604 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.604 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.604 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339670+0000", 2024-04-26T01:08:31.604 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000113047 2024-04-26T01:08:31.604 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.604 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.604 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.604 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.604 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpRead(3.2s2 33/30 ECSubRead(tid=161, to_read={3:49548768:::smithi16428812-5:head=0,4096,0}, subchunks={3:49548768:::smithi16428812-5:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:31.604 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.337412+0000", 2024-04-26T01:08:31.604 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.218627689000002, 2024-04-26T01:08:31.604 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0052553979999999997, 2024-04-26T01:08:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337412+0000", 2024-04-26T01:08:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337412+0000", 2024-04-26T01:08:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.606 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.606 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.606 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337413+0000", 2024-04-26T01:08:31.606 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2049999999999999e-06 2024-04-26T01:08:31.606 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.606 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.606 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337432+0000", 2024-04-26T01:08:31.606 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8272000000000001e-05 2024-04-26T01:08:31.606 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.607 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.607 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341917+0000", 2024-04-26T01:08:31.607 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0044857359999999997 2024-04-26T01:08:31.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.607 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.607 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341951+0000", 2024-04-26T01:08:31.607 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3173000000000002e-05 2024-04-26T01:08:31.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.607 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.607 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.342131+0000", 2024-04-26T01:08:31.607 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000180457 2024-04-26T01:08:31.608 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.608 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.608 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.608 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.342668+0000", 2024-04-26T01:08:31.608 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00053655499999999997 2024-04-26T01:08:31.608 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.608 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.608 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.608 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.608 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.608 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpRead(3.2s2 33/30 ECSubRead(tid=163, to_read={3:410d186f:::smithi16428812-11:head=0,4096,0}, subchunks={3:410d186f:::smithi16428812-11:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:31.608 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.342255+0000", 2024-04-26T01:08:31.608 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.213785225999999, 2024-04-26T01:08:31.608 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.047919758, 2024-04-26T01:08:31.609 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.609 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.609 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.609 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.609 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.609 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.609 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.342255+0000", 2024-04-26T01:08:31.609 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.609 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.609 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.609 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.609 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.342255+0000", 2024-04-26T01:08:31.609 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.610 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.610 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.342256+0000", 2024-04-26T01:08:31.610 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3969999999999999e-06 2024-04-26T01:08:31.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.610 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.610 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.342290+0000", 2024-04-26T01:08:31.610 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4085000000000003e-05 2024-04-26T01:08:31.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.611 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.611 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.611 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.389563+0000", 2024-04-26T01:08:31.611 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.047272886 2024-04-26T01:08:31.611 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.611 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.611 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.611 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.389578+0000", 2024-04-26T01:08:31.611 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4678000000000001e-05 2024-04-26T01:08:31.611 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.611 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.611 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.612 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.389769+0000", 2024-04-26T01:08:31.612 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019143099999999999 2024-04-26T01:08:31.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.612 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.612 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.390174+0000", 2024-04-26T01:08:31.612 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00040528100000000001 2024-04-26T01:08:31.612 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.612 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.612 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.613 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/30 ECSubWrite(tid=164, reqid=client.4521.0:79, at_version=33'5, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:08:31.613 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.389336+0000", 2024-04-26T01:08:31.613 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.166704207000002, 2024-04-26T01:08:31.613 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0067203230000000003, 2024-04-26T01:08:31.613 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.613 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.613 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.613 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.613 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.613 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.389336+0000", 2024-04-26T01:08:31.613 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.613 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.614 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.614 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.389336+0000", 2024-04-26T01:08:31.614 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.614 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.614 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.614 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.614 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.389341+0000", 2024-04-26T01:08:31.614 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9679999999999997e-06 2024-04-26T01:08:31.614 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.614 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.614 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.615 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.389515+0000", 2024-04-26T01:08:31.615 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017436400000000001 2024-04-26T01:08:31.615 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.615 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.615 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.615 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394634+0000", 2024-04-26T01:08:31.615 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.005119513 2024-04-26T01:08:31.615 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.615 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.615 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.615 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394651+0000", 2024-04-26T01:08:31.615 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6603999999999999e-05 2024-04-26T01:08:31.615 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.616 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.616 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.616 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394757+0000", 2024-04-26T01:08:31.616 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000105531 2024-04-26T01:08:31.616 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.616 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.616 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.616 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394769+0000", 2024-04-26T01:08:31.616 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2223e-05 2024-04-26T01:08:31.616 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.616 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.616 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.617 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.396012+0000", 2024-04-26T01:08:31.617 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012432700000000001 2024-04-26T01:08:31.617 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.617 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.617 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.617 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.396056+0000", 2024-04-26T01:08:31.617 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3850000000000002e-05 2024-04-26T01:08:31.617 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.617 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.617 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.617 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.617 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.617 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/30 ECSubWrite(tid=165, reqid=client.4521.0:80, at_version=33'6, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:08:31.618 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.394557+0000", 2024-04-26T01:08:31.618 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.161482839999998, 2024-04-26T01:08:31.618 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.039358791999999997, 2024-04-26T01:08:31.618 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.618 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.618 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.618 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.618 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.618 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.618 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394557+0000", 2024-04-26T01:08:31.618 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.618 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.619 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.619 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.619 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394557+0000", 2024-04-26T01:08:31.619 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.619 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.619 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.619 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394559+0000", 2024-04-26T01:08:31.619 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8500000000000001e-06 2024-04-26T01:08:31.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.619 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.619 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.620 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394591+0000", 2024-04-26T01:08:31.620 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1967000000000001e-05 2024-04-26T01:08:31.620 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.620 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.620 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.620 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432679+0000", 2024-04-26T01:08:31.620 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.038087835 2024-04-26T01:08:31.620 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.620 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.620 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.620 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432698+0000", 2024-04-26T01:08:31.620 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9459000000000001e-05 2024-04-26T01:08:31.621 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.621 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.621 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.621 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432802+0000", 2024-04-26T01:08:31.621 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000103404 2024-04-26T01:08:31.621 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.621 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.621 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.621 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432812+0000", 2024-04-26T01:08:31.621 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0969e-05 2024-04-26T01:08:31.621 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.622 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.622 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.433867+0000", 2024-04-26T01:08:31.622 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010541019999999999 2024-04-26T01:08:31.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.622 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.622 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.433916+0000", 2024-04-26T01:08:31.622 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9206000000000003e-05 2024-04-26T01:08:31.622 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.622 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.622 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.623 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.623 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.623 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.3s6 33/30 ECSubWrite(tid=166, reqid=client.4521.0:109, at_version=33'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:31.623 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.432104+0000", 2024-04-26T01:08:31.623 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.123935725000003, 2024-04-26T01:08:31.623 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042723700000000002, 2024-04-26T01:08:31.623 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.623 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.623 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.623 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.623 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.623 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.624 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432104+0000", 2024-04-26T01:08:31.624 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.624 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.624 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432104+0000", 2024-04-26T01:08:31.624 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.624 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.624 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432106+0000", 2024-04-26T01:08:31.624 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1830000000000001e-06 2024-04-26T01:08:31.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.625 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.625 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.625 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.432629+0000", 2024-04-26T01:08:31.625 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00052224499999999996 2024-04-26T01:08:31.625 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.625 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.625 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.625 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434348+0000", 2024-04-26T01:08:31.625 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001719583 2024-04-26T01:08:31.625 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.625 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.626 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.626 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434358+0000", 2024-04-26T01:08:31.626 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6930000000000002e-06 2024-04-26T01:08:31.626 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.626 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.626 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.626 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434474+0000", 2024-04-26T01:08:31.626 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011661400000000001 2024-04-26T01:08:31.626 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.626 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.626 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.627 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434507+0000", 2024-04-26T01:08:31.627 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.2957000000000003e-05 2024-04-26T01:08:31.627 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.627 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.627 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.436338+0000", 2024-04-26T01:08:31.627 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0018304300000000001 2024-04-26T01:08:31.627 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.627 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.627 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.436376+0000", 2024-04-26T01:08:31.627 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8665000000000003e-05 2024-04-26T01:08:31.627 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.628 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.628 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.628 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.628 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.628 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.3s6 33/30 ECSubWrite(tid=167, reqid=client.4521.0:110, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:31.628 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.434303+0000", 2024-04-26T01:08:31.628 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.121737244999998, 2024-04-26T01:08:31.628 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0051688970000000004, 2024-04-26T01:08:31.628 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.628 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.628 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.628 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.629 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.629 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434303+0000", 2024-04-26T01:08:31.629 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.629 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.629 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434303+0000", 2024-04-26T01:08:31.629 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.629 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.630 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434303+0000", 2024-04-26T01:08:31.630 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7499999999999999e-07 2024-04-26T01:08:31.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.630 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.630 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.434327+0000", 2024-04-26T01:08:31.630 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3431e-05 2024-04-26T01:08:31.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.630 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.630 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.438328+0000", 2024-04-26T01:08:31.630 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0040014600000000001 2024-04-26T01:08:31.631 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.631 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.631 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.631 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.438338+0000", 2024-04-26T01:08:31.631 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7799999999999995e-06 2024-04-26T01:08:31.631 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.631 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.631 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.631 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.438415+0000", 2024-04-26T01:08:31.631 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6911999999999998e-05 2024-04-26T01:08:31.631 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.631 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.631 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.632 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.438428+0000", 2024-04-26T01:08:31.632 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3465000000000001e-05 2024-04-26T01:08:31.632 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.632 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.632 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.632 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.439422+0000", 2024-04-26T01:08:31.632 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00099357200000000007 2024-04-26T01:08:31.632 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.632 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.632 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.632 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.439472+0000", 2024-04-26T01:08:31.632 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9502000000000001e-05 2024-04-26T01:08:31.632 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.633 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.633 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.633 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.633 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.633 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.3s6 33/30 ECSubWrite(tid=170, reqid=client.4521.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=33'2))", 2024-04-26T01:08:31.633 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.442037+0000", 2024-04-26T01:08:31.633 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.114002773000003, 2024-04-26T01:08:31.633 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.064321377999999998, 2024-04-26T01:08:31.633 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.633 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.633 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.633 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.633 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.634 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.634 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.442037+0000", 2024-04-26T01:08:31.634 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.634 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.634 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.634 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.634 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.442037+0000", 2024-04-26T01:08:31.634 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.634 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.634 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.634 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.634 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.442038+0000", 2024-04-26T01:08:31.635 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9800000000000002e-07 2024-04-26T01:08:31.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.635 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.635 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.635 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.442053+0000", 2024-04-26T01:08:31.635 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4816e-05 2024-04-26T01:08:31.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.635 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.635 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.635 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.505728+0000", 2024-04-26T01:08:31.635 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.063674713999999993 2024-04-26T01:08:31.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.636 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.636 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.636 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.505736+0000", 2024-04-26T01:08:31.636 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.174e-06 2024-04-26T01:08:31.636 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.636 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.636 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.636 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.505807+0000", 2024-04-26T01:08:31.636 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1203000000000003e-05 2024-04-26T01:08:31.636 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.636 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.636 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.637 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.505816+0000", 2024-04-26T01:08:31.637 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1840000000000005e-06 2024-04-26T01:08:31.637 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.637 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.637 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.637 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.506316+0000", 2024-04-26T01:08:31.637 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00050021600000000005 2024-04-26T01:08:31.637 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.637 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.637 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.637 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.506358+0000", 2024-04-26T01:08:31.637 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2073000000000002e-05 2024-04-26T01:08:31.638 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.638 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.638 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.638 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.638 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 36/30 ECSubWrite(tid=184, reqid=client.4521.0:225, at_version=36'13, trim_to=0'0, roll_forward_to=33'10))", 2024-04-26T01:08:31.638 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:02.440081+0000", 2024-04-26T01:08:31.638 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 29.115958762999998, 2024-04-26T01:08:31.638 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0063574779999999997, 2024-04-26T01:08:31.638 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.638 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.638 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.638 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.639 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.639 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.440081+0000", 2024-04-26T01:08:31.639 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.639 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.639 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.639 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.440081+0000", 2024-04-26T01:08:31.639 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.639 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.639 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.639 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.440081+0000", 2024-04-26T01:08:31.639 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9700000000000002e-07 2024-04-26T01:08:31.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.640 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.440123+0000", 2024-04-26T01:08:31.640 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1319000000000002e-05 2024-04-26T01:08:31.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.640 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.440215+0000", 2024-04-26T01:08:31.640 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.2264999999999998e-05 2024-04-26T01:08:31.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.641 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.641 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.440238+0000", 2024-04-26T01:08:31.641 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2840000000000002e-05 2024-04-26T01:08:31.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.641 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.641 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.641 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.441207+0000", 2024-04-26T01:08:31.641 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00096928499999999996 2024-04-26T01:08:31.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.641 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.641 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.641 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.441230+0000", 2024-04-26T01:08:31.642 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3003e-05 2024-04-26T01:08:31.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.642 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.642 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.446356+0000", 2024-04-26T01:08:31.642 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0051255169999999996 2024-04-26T01:08:31.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.642 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.642 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.446439+0000", 2024-04-26T01:08:31.642 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2851999999999999e-05 2024-04-26T01:08:31.643 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.643 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.643 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.643 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s2 36/30 ECSubWrite(tid=185, reqid=client.4521.0:225, at_version=0'0, trim_to=0'0, roll_forward_to=36'13))", 2024-04-26T01:08:31.643 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:02.447653+0000", 2024-04-26T01:08:31.643 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 29.108387161, 2024-04-26T01:08:31.643 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0046212900000000001, 2024-04-26T01:08:31.643 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.643 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.644 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.644 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.644 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.644 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.447653+0000", 2024-04-26T01:08:31.644 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.644 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.644 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.447653+0000", 2024-04-26T01:08:31.644 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.645 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.645 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.447655+0000", 2024-04-26T01:08:31.645 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9360000000000002e-06 2024-04-26T01:08:31.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.645 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.645 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.447684+0000", 2024-04-26T01:08:31.645 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9713000000000002e-05 2024-04-26T01:08:31.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.646 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.646 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.447811+0000", 2024-04-26T01:08:31.646 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012631099999999999 2024-04-26T01:08:31.646 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.646 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.646 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.646 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.447837+0000", 2024-04-26T01:08:31.646 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6016999999999998e-05 2024-04-26T01:08:31.646 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.646 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.646 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.646 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.448130+0000", 2024-04-26T01:08:31.647 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00029376000000000001 2024-04-26T01:08:31.647 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.647 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.647 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.647 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.448159+0000", 2024-04-26T01:08:31.647 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8802999999999998e-05 2024-04-26T01:08:31.647 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.647 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.647 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.647 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.452179+0000", 2024-04-26T01:08:31.647 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0040198259999999998 2024-04-26T01:08:31.647 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.648 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.648 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.648 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:02.452274+0000", 2024-04-26T01:08:31.648 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.4924000000000001e-05 2024-04-26T01:08:31.648 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.648 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.648 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.648 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.648 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.648 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(osd.4.21:6 3.as0 3:592ac288:::smithi16428812-20:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-26T01:08:31.648 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:19.119924+0000", 2024-04-26T01:08:31.648 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 12.436116113000001, 2024-04-26T01:08:31.649 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0081993280000000005, 2024-04-26T01:08:31.649 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.649 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.649 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:31.649 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.649 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.649 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.649 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:19.119924+0000", 2024-04-26T01:08:31.649 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.649 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.649 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.649 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.649 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:19.119924+0000", 2024-04-26T01:08:31.650 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.650 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.650 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.650 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.650 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:19.119926+0000", 2024-04-26T01:08:31.650 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5000000000000002e-06 2024-04-26T01:08:31.650 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.650 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.650 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.650 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:19.119938+0000", 2024-04-26T01:08:31.650 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1782e-05 2024-04-26T01:08:31.650 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.651 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.651 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:19.119972+0000", 2024-04-26T01:08:31.651 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3667000000000001e-05 2024-04-26T01:08:31.651 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.651 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.651 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:19.119998+0000", 2024-04-26T01:08:31.651 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6069000000000001e-05 2024-04-26T01:08:31.651 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.651 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.652 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:19.120205+0000", 2024-04-26T01:08:31.652 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00020710100000000001 2024-04-26T01:08:31.652 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.652 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.652 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2024-04-26T01:08:31.652 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:19.120221+0000", 2024-04-26T01:08:31.652 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6582000000000001e-05 2024-04-26T01:08:31.652 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.652 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.652 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.652 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:19.123766+0000", 2024-04-26T01:08:31.652 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0035442350000000002 2024-04-26T01:08:31.653 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.653 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.653 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:31.653 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:19.124009+0000", 2024-04-26T01:08:31.653 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000243136 2024-04-26T01:08:31.653 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.653 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.653 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.653 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:19.128123+0000", 2024-04-26T01:08:31.653 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0041142560000000002 2024-04-26T01:08:31.653 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.653 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.653 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.654 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.654 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.654 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.3s6 63/30 ECSubWrite(tid=223, reqid=client.4521.0:675, at_version=63'15, trim_to=0'0, roll_forward_to=44'13))", 2024-04-26T01:08:31.654 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:29.753306+0000", 2024-04-26T01:08:31.654 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 1.802733519, 2024-04-26T01:08:31.654 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0049703869999999997, 2024-04-26T01:08:31.654 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:31.654 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:31.654 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:31.654 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:31.654 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.654 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:31.655 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:29.753306+0000", 2024-04-26T01:08:31.655 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.655 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.655 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.655 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:31.655 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:29.753306+0000", 2024-04-26T01:08:31.655 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:31.655 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.655 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.655 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:31.655 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:29.753308+0000", 2024-04-26T01:08:31.655 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7400000000000001e-06 2024-04-26T01:08:31.656 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.656 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.656 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:31.656 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:29.753659+0000", 2024-04-26T01:08:31.656 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00035091499999999998 2024-04-26T01:08:31.656 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.656 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.656 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:31.656 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:29.753746+0000", 2024-04-26T01:08:31.656 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.6922000000000006e-05 2024-04-26T01:08:31.656 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.656 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.656 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:31.657 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:29.753762+0000", 2024-04-26T01:08:31.657 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6073e-05 2024-04-26T01:08:31.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.657 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:31.657 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:29.753884+0000", 2024-04-26T01:08:31.657 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000121678 2024-04-26T01:08:31.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.657 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:31.657 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:29.753903+0000", 2024-04-26T01:08:31.657 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9551e-05 2024-04-26T01:08:31.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.658 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:31.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:29.758221+0000", 2024-04-26T01:08:31.658 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0043177090000000003 2024-04-26T01:08:31.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:31.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:31.658 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:31.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:29.758277+0000", 2024-04-26T01:08:31.658 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5798999999999999e-05 2024-04-26T01:08:31.658 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.658 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.658 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.659 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:31.659 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:31.659 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:31.660 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:08:31.661 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:31.657+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:31.761 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:31.759+0000 7ff2871de640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 32810) UID: 0 2024-04-26T01:08:31.787 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:31.788 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:31.788 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:08:31.788 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:31.801 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:08:31.862 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:31.861+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:31.963 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:31.960+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:31.969 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:31.969 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:31.969 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2024-04-26T01:08:31.969 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2024-04-26T01:08:31.969 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:31.983 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:08:32.040 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.1 to 1 (10000) 2024-04-26T01:08:32.045 INFO:tasks.rados.rados.0.smithi164.stdout:274: done (2 left) 2024-04-26T01:08:32.045 INFO:tasks.rados.rados.0.smithi164.stdout:275: snap_remove snap 15 2024-04-26T01:08:32.064 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:32.063+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:32.065 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-26T01:08:32.106 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:32.106 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2024-04-26T01:08:32.106 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2024-04-26T01:08:32.107 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2024-04-26T01:08:32.107 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.107 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:4 3.ds0 3:bcd9cb81:::smithi16428812-2:head [append 0~32768,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:32.107 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.302987+0000", 2024-04-26T01:08:32.107 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.803213786000001, 2024-04-26T01:08:32.107 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012685460000000001, 2024-04-26T01:08:32.107 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.107 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.107 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.107 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.107 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.107 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.107 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 4 2024-04-26T01:08:32.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.108 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.108 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.108 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.108 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.302987+0000", 2024-04-26T01:08:32.108 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.108 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.108 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.108 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.302987+0000", 2024-04-26T01:08:32.108 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.108 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.109 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.109 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.302989+0000", 2024-04-26T01:08:32.109 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.266e-06 2024-04-26T01:08:32.109 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.109 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.109 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.109 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303040+0000", 2024-04-26T01:08:32.109 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1029000000000002e-05 2024-04-26T01:08:32.109 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.109 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.109 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.109 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303059+0000", 2024-04-26T01:08:32.109 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8909000000000001e-05 2024-04-26T01:08:32.109 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.110 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.110 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.110 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303074+0000", 2024-04-26T01:08:32.110 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4812e-05 2024-04-26T01:08:32.110 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.110 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.110 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.110 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303197+0000", 2024-04-26T01:08:32.110 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000122748 2024-04-26T01:08:32.110 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.110 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.110 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.110 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303424+0000", 2024-04-26T01:08:32.110 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022748500000000001 2024-04-26T01:08:32.111 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.111 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.111 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.111 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304556+0000", 2024-04-26T01:08:32.111 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001131948 2024-04-26T01:08:32.111 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.111 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.111 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.111 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.313947+0000", 2024-04-26T01:08:32.111 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0093914069999999992 2024-04-26T01:08:32.111 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.111 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.112 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.112 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315465+0000", 2024-04-26T01:08:32.112 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001517673 2024-04-26T01:08:32.112 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.112 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.112 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.112 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315672+0000", 2024-04-26T01:08:32.112 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000207183 2024-04-26T01:08:32.112 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.112 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.112 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.112 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.112 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.112 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:5 3.ds0 3:bcd9cb81:::smithi16428812-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:32.113 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.303108+0000", 2024-04-26T01:08:32.113 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.803092788000001, 2024-04-26T01:08:32.113 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013204789, 2024-04-26T01:08:32.113 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.113 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.113 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.113 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.113 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.113 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.113 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 5 2024-04-26T01:08:32.113 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.113 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.113 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.114 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.114 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303108+0000", 2024-04-26T01:08:32.114 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.114 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.114 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.114 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.114 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303108+0000", 2024-04-26T01:08:32.114 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.114 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.114 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.114 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.114 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303109+0000", 2024-04-26T01:08:32.114 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3149999999999999e-06 2024-04-26T01:08:32.114 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.115 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.115 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.115 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303117+0000", 2024-04-26T01:08:32.115 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1899999999999995e-06 2024-04-26T01:08:32.115 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.115 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.115 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.115 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303131+0000", 2024-04-26T01:08:32.115 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3890000000000001e-05 2024-04-26T01:08:32.115 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.115 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.115 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.115 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303143+0000", 2024-04-26T01:08:32.115 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1773999999999999e-05 2024-04-26T01:08:32.116 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.116 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.116 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.116 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.305710+0000", 2024-04-26T01:08:32.116 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0025668930000000002 2024-04-26T01:08:32.116 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.116 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.116 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.116 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.305934+0000", 2024-04-26T01:08:32.116 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022386800000000001 2024-04-26T01:08:32.116 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.116 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.116 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.116 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.307007+0000", 2024-04-26T01:08:32.117 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010735930000000001 2024-04-26T01:08:32.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.117 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.117 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.313953+0000", 2024-04-26T01:08:32.117 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006945657 2024-04-26T01:08:32.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.117 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.117 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316089+0000", 2024-04-26T01:08:32.117 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021360509999999999 2024-04-26T01:08:32.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.118 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.118 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316313+0000", 2024-04-26T01:08:32.118 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000223558 2024-04-26T01:08:32.118 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.118 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.118 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.118 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:6 3.ds0 3:bcd9cb81:::smithi16428812-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:32.118 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.303162+0000", 2024-04-26T01:08:32.118 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.803038806000004, 2024-04-26T01:08:32.118 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.027157324, 2024-04-26T01:08:32.118 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.119 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.119 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:32.119 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.119 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.119 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.119 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 6 2024-04-26T01:08:32.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.119 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.119 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.119 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303162+0000", 2024-04-26T01:08:32.119 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.120 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.120 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303162+0000", 2024-04-26T01:08:32.120 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.120 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.120 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303163+0000", 2024-04-26T01:08:32.120 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.145e-06 2024-04-26T01:08:32.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.120 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.120 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303169+0000", 2024-04-26T01:08:32.120 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8150000000000002e-06 2024-04-26T01:08:32.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.121 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.121 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303184+0000", 2024-04-26T01:08:32.121 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5116000000000001e-05 2024-04-26T01:08:32.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.121 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.121 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303196+0000", 2024-04-26T01:08:32.121 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1778e-05 2024-04-26T01:08:32.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.121 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.121 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308497+0000", 2024-04-26T01:08:32.122 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0053009449999999996 2024-04-26T01:08:32.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.122 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:32.122 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308937+0000", 2024-04-26T01:08:32.122 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00044013100000000002 2024-04-26T01:08:32.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.122 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.122 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.317636+0000", 2024-04-26T01:08:32.122 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0086997570000000007 2024-04-26T01:08:32.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.122 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.123 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.317809+0000", 2024-04-26T01:08:32.123 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017204899999999999 2024-04-26T01:08:32.123 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.123 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.123 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.330319+0000", 2024-04-26T01:08:32.123 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012510588 2024-04-26T01:08:32.123 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.123 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.123 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.123 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.123 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 33/30 ECSubWriteReply(tid=1, last_complete=32'2, committed=1, applied=1))", 2024-04-26T01:08:32.123 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.309264+0000", 2024-04-26T01:08:32.124 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.796936588999998, 2024-04-26T01:08:32.124 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0055114789999999997, 2024-04-26T01:08:32.124 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.124 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.124 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:32.124 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.124 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.124 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.124 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309264+0000", 2024-04-26T01:08:32.124 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.124 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.124 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.124 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.124 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309264+0000", 2024-04-26T01:08:32.125 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.125 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.125 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.125 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.125 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309266+0000", 2024-04-26T01:08:32.125 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8369999999999999e-06 2024-04-26T01:08:32.125 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.125 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.125 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.125 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309277+0000", 2024-04-26T01:08:32.125 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0991e-05 2024-04-26T01:08:32.125 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.125 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.125 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.125 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309352+0000", 2024-04-26T01:08:32.126 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4901999999999995e-05 2024-04-26T01:08:32.126 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.126 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.126 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.126 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309371+0000", 2024-04-26T01:08:32.126 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8828e-05 2024-04-26T01:08:32.126 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.126 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.126 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.126 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314745+0000", 2024-04-26T01:08:32.126 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0053747309999999998 2024-04-26T01:08:32.126 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.126 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.127 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.127 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314775+0000", 2024-04-26T01:08:32.127 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0190000000000001e-05 2024-04-26T01:08:32.127 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.127 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.127 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.127 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.127 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.127 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 32/30 ECSubWriteReply(tid=1, last_complete=32'2, committed=1, applied=1))", 2024-04-26T01:08:32.127 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.309299+0000", 2024-04-26T01:08:32.127 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.796901269999999, 2024-04-26T01:08:32.127 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056645660000000002, 2024-04-26T01:08:32.127 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.127 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.128 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:32.128 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.128 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.128 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.128 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309299+0000", 2024-04-26T01:08:32.128 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.128 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.128 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.128 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.128 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309299+0000", 2024-04-26T01:08:32.128 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.128 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.128 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.129 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.129 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309300+0000", 2024-04-26T01:08:32.129 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0300000000000001e-06 2024-04-26T01:08:32.129 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.129 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.129 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.129 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309308+0000", 2024-04-26T01:08:32.129 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9629999999999998e-06 2024-04-26T01:08:32.129 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.129 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.129 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.130 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309369+0000", 2024-04-26T01:08:32.130 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0223000000000003e-05 2024-04-26T01:08:32.130 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.130 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.130 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.130 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.309383+0000", 2024-04-26T01:08:32.130 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4728e-05 2024-04-26T01:08:32.130 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.130 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.130 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.130 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314927+0000", 2024-04-26T01:08:32.130 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0055432650000000003 2024-04-26T01:08:32.130 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.130 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.131 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.131 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.314964+0000", 2024-04-26T01:08:32.131 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7357000000000002e-05 2024-04-26T01:08:32.131 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.131 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.131 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.131 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.131 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.131 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:34 3.ds0 3:b1366e28:::smithi16428812-12:head [append 0~196608,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:32.131 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321139+0000", 2024-04-26T01:08:32.131 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.785061568000003, 2024-04-26T01:08:32.131 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015400954999999999, 2024-04-26T01:08:32.132 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.132 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.132 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.132 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.132 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.132 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.132 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 34 2024-04-26T01:08:32.132 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.132 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.132 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.132 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.132 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321139+0000", 2024-04-26T01:08:32.132 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.132 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.133 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.133 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.133 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321139+0000", 2024-04-26T01:08:32.133 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.133 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.133 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.133 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.133 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321142+0000", 2024-04-26T01:08:32.133 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8660000000000002e-06 2024-04-26T01:08:32.133 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.133 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.133 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.133 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321475+0000", 2024-04-26T01:08:32.133 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00033334599999999999 2024-04-26T01:08:32.133 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.134 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.134 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.134 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321504+0000", 2024-04-26T01:08:32.134 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8445e-05 2024-04-26T01:08:32.134 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.134 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.134 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.134 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321518+0000", 2024-04-26T01:08:32.134 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4708e-05 2024-04-26T01:08:32.134 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.134 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.134 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.134 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322532+0000", 2024-04-26T01:08:32.135 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001013246 2024-04-26T01:08:32.135 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.135 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.135 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.135 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322688+0000", 2024-04-26T01:08:32.135 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000156525 2024-04-26T01:08:32.135 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.135 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.135 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.135 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323904+0000", 2024-04-26T01:08:32.135 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012160160000000001 2024-04-26T01:08:32.135 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.135 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.135 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.332759+0000", 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0088551199999999993 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336395+0000", 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0036352559999999999 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336540+0000", 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000145427 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.136 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.137 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.137 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:35 3.ds0 3:b1366e28:::smithi16428812-12:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:32.137 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321544+0000", 2024-04-26T01:08:32.137 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.78465645, 2024-04-26T01:08:32.137 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015414796, 2024-04-26T01:08:32.137 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.137 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.137 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.137 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.137 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.137 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.137 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 35 2024-04-26T01:08:32.137 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.137 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.137 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.138 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.138 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321544+0000", 2024-04-26T01:08:32.138 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.138 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.138 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.138 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.138 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321544+0000", 2024-04-26T01:08:32.138 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.138 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.138 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.138 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.138 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321545+0000", 2024-04-26T01:08:32.138 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3340000000000001e-06 2024-04-26T01:08:32.139 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.139 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.139 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.139 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321553+0000", 2024-04-26T01:08:32.139 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4739999999999999e-06 2024-04-26T01:08:32.139 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.139 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.139 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.139 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321571+0000", 2024-04-26T01:08:32.139 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7688000000000001e-05 2024-04-26T01:08:32.139 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.139 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.139 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.139 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321614+0000", 2024-04-26T01:08:32.140 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3047000000000003e-05 2024-04-26T01:08:32.140 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.140 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.140 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.140 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324961+0000", 2024-04-26T01:08:32.140 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003347184 2024-04-26T01:08:32.140 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.140 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.140 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.140 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.325113+0000", 2024-04-26T01:08:32.140 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000152287 2024-04-26T01:08:32.140 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.141 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.141 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.141 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.327055+0000", 2024-04-26T01:08:32.141 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001942292 2024-04-26T01:08:32.141 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.141 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.141 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.141 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.332763+0000", 2024-04-26T01:08:32.141 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0057070819999999996 2024-04-26T01:08:32.141 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.141 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.141 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.141 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336824+0000", 2024-04-26T01:08:32.141 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0040615449999999997 2024-04-26T01:08:32.141 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.142 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.142 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.142 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336959+0000", 2024-04-26T01:08:32.142 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013486300000000001 2024-04-26T01:08:32.142 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.142 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.142 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.142 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.142 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.142 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:36 3.ds0 3:b1366e28:::smithi16428812-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:32.142 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321633+0000", 2024-04-26T01:08:32.142 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.784567613, 2024-04-26T01:08:32.142 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.019982366000000001, 2024-04-26T01:08:32.142 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.143 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.143 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:32.143 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.143 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.143 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.143 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 36 2024-04-26T01:08:32.143 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.143 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.143 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.143 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.143 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321633+0000", 2024-04-26T01:08:32.143 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.143 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.143 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.144 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.144 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321633+0000", 2024-04-26T01:08:32.144 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.144 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.144 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.144 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.144 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321634+0000", 2024-04-26T01:08:32.144 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.268e-06 2024-04-26T01:08:32.144 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.144 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.144 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.144 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321641+0000", 2024-04-26T01:08:32.144 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5100000000000004e-06 2024-04-26T01:08:32.144 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.145 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.145 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.145 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321656+0000", 2024-04-26T01:08:32.145 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5191000000000001e-05 2024-04-26T01:08:32.145 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.145 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.145 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.145 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321666+0000", 2024-04-26T01:08:32.145 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0178e-05 2024-04-26T01:08:32.145 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.145 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.145 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.145 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.333161+0000", 2024-04-26T01:08:32.145 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011494605999999999 2024-04-26T01:08:32.145 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.146 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.146 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:32.146 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.333311+0000", 2024-04-26T01:08:32.146 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015046100000000001 2024-04-26T01:08:32.146 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.146 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.146 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.146 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338017+0000", 2024-04-26T01:08:32.146 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0047055489999999998 2024-04-26T01:08:32.146 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.146 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.146 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.146 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338172+0000", 2024-04-26T01:08:32.147 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015476500000000001 2024-04-26T01:08:32.147 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.147 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.147 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.147 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341615+0000", 2024-04-26T01:08:32.147 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0034438379999999998 2024-04-26T01:08:32.147 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.147 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.147 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.147 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.147 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.148 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpReadReply(3.ds0 33/30 ECSubReadReply(tid=3, attrs_read=0))", 2024-04-26T01:08:32.148 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.322592+0000", 2024-04-26T01:08:32.148 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.783608602000001, 2024-04-26T01:08:32.148 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077557909999999997, 2024-04-26T01:08:32.148 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.148 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.148 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:32.148 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.149 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.149 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.149 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322592+0000", 2024-04-26T01:08:32.149 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.149 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.149 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.149 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.149 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322592+0000", 2024-04-26T01:08:32.149 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.149 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.149 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.149 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.149 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322594+0000", 2024-04-26T01:08:32.149 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7740000000000001e-06 2024-04-26T01:08:32.150 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.150 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.150 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.150 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322617+0000", 2024-04-26T01:08:32.150 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3074999999999998e-05 2024-04-26T01:08:32.150 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.150 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.150 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.150 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322695+0000", 2024-04-26T01:08:32.150 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7892999999999994e-05 2024-04-26T01:08:32.150 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.150 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.150 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.151 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322710+0000", 2024-04-26T01:08:32.151 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.505e-05 2024-04-26T01:08:32.151 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.151 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.151 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.151 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329473+0000", 2024-04-26T01:08:32.151 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006763682 2024-04-26T01:08:32.151 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.151 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.151 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.151 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.330348+0000", 2024-04-26T01:08:32.151 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000874317 2024-04-26T01:08:32.151 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.151 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.152 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.152 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.152 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.152 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 32/30 ECSubWriteReply(tid=2, last_complete=32'2, committed=1, applied=1))", 2024-04-26T01:08:32.152 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.323325+0000", 2024-04-26T01:08:32.152 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.782875265000001, 2024-04-26T01:08:32.152 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0060949649999999999, 2024-04-26T01:08:32.152 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.152 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.152 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:32.152 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.152 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.153 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.153 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323325+0000", 2024-04-26T01:08:32.153 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.153 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.153 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.153 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.153 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323325+0000", 2024-04-26T01:08:32.153 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.153 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.153 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.153 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.153 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323327+0000", 2024-04-26T01:08:32.153 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.195e-06 2024-04-26T01:08:32.153 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.154 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.154 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.154 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323334+0000", 2024-04-26T01:08:32.154 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2130000000000002e-06 2024-04-26T01:08:32.154 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.154 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.154 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.154 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323418+0000", 2024-04-26T01:08:32.154 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4499999999999994e-05 2024-04-26T01:08:32.154 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.154 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.154 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.154 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323434+0000", 2024-04-26T01:08:32.154 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5338000000000001e-05 2024-04-26T01:08:32.155 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.155 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.155 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.155 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329342+0000", 2024-04-26T01:08:32.155 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0059083399999999998 2024-04-26T01:08:32.155 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.155 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.155 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.155 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329420+0000", 2024-04-26T01:08:32.155 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8379000000000003e-05 2024-04-26T01:08:32.155 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.155 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.155 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.155 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.155 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.156 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 32/30 ECSubWrite(tid=0, reqid=client.4521.0:28, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:32.156 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.326088+0000", 2024-04-26T01:08:32.156 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.780112766999999, 2024-04-26T01:08:32.156 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0097951900000000005, 2024-04-26T01:08:32.156 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.156 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.156 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:32.156 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.156 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.156 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.156 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326088+0000", 2024-04-26T01:08:32.156 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.156 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.156 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.157 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.157 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326088+0000", 2024-04-26T01:08:32.157 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.157 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.157 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.157 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.157 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326089+0000", 2024-04-26T01:08:32.157 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5460000000000001e-06 2024-04-26T01:08:32.157 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.157 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.157 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.157 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326154+0000", 2024-04-26T01:08:32.157 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4979000000000002e-05 2024-04-26T01:08:32.157 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.157 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.158 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.158 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326251+0000", 2024-04-26T01:08:32.158 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7139999999999995e-05 2024-04-26T01:08:32.158 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.158 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.158 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.158 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326266+0000", 2024-04-26T01:08:32.158 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4355999999999999e-05 2024-04-26T01:08:32.158 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.158 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.158 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.158 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.330888+0000", 2024-04-26T01:08:32.158 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0046226310000000003 2024-04-26T01:08:32.158 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.159 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.159 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.159 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.331237+0000", 2024-04-26T01:08:32.159 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000348103 2024-04-26T01:08:32.159 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.159 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.159 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.159 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335801+0000", 2024-04-26T01:08:32.159 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0045642189999999996 2024-04-26T01:08:32.159 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.159 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.159 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.159 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335883+0000", 2024-04-26T01:08:32.159 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2216e-05 2024-04-26T01:08:32.160 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.160 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.160 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.160 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.160 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.160 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:46 3.ds0 3:b85e6c81:::smithi16428812-16:head [append 0~262144,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:32.160 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.338039+0000", 2024-04-26T01:08:32.160 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.768161679000002, 2024-04-26T01:08:32.160 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0092704039999999995, 2024-04-26T01:08:32.160 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.160 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.160 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.160 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.161 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.161 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.161 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 46 2024-04-26T01:08:32.161 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.161 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.161 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.161 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.161 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338039+0000", 2024-04-26T01:08:32.161 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.161 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.161 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.161 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.161 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338039+0000", 2024-04-26T01:08:32.161 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.162 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.162 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.162 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.162 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338041+0000", 2024-04-26T01:08:32.162 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4660000000000002e-06 2024-04-26T01:08:32.162 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.162 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.162 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.162 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338148+0000", 2024-04-26T01:08:32.162 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000106384 2024-04-26T01:08:32.162 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.162 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.162 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.163 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338170+0000", 2024-04-26T01:08:32.163 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2022e-05 2024-04-26T01:08:32.163 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.163 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.163 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.163 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338184+0000", 2024-04-26T01:08:32.163 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4345000000000001e-05 2024-04-26T01:08:32.163 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.163 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.163 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.163 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341687+0000", 2024-04-26T01:08:32.163 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003502631 2024-04-26T01:08:32.164 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.164 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.164 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.164 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341929+0000", 2024-04-26T01:08:32.164 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024248 2024-04-26T01:08:32.164 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.164 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.164 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.164 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.343657+0000", 2024-04-26T01:08:32.164 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001727302 2024-04-26T01:08:32.165 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.165 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.165 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.165 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346591+0000", 2024-04-26T01:08:32.165 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0029344420000000002 2024-04-26T01:08:32.165 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.165 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.165 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.165 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.347148+0000", 2024-04-26T01:08:32.165 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000556801 2024-04-26T01:08:32.165 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.165 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.165 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.165 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.347309+0000", 2024-04-26T01:08:32.166 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000161531 2024-04-26T01:08:32.166 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.166 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.166 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.166 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.166 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.166 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:47 3.ds0 3:b85e6c81:::smithi16428812-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:32.166 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.338373+0000", 2024-04-26T01:08:32.166 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.767827996999998, 2024-04-26T01:08:32.166 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012969223, 2024-04-26T01:08:32.166 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.166 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.166 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.166 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.167 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.167 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.167 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 47 2024-04-26T01:08:32.167 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.167 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.167 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.167 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.167 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338373+0000", 2024-04-26T01:08:32.167 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.167 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.167 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.167 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.167 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338373+0000", 2024-04-26T01:08:32.167 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.168 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.168 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.168 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.168 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338374+0000", 2024-04-26T01:08:32.168 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.782e-06 2024-04-26T01:08:32.168 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.168 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.168 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.168 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338383+0000", 2024-04-26T01:08:32.168 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.6570000000000003e-06 2024-04-26T01:08:32.168 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.168 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.169 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.169 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338402+0000", 2024-04-26T01:08:32.169 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8708000000000001e-05 2024-04-26T01:08:32.169 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.169 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.169 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.169 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338415+0000", 2024-04-26T01:08:32.169 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2976000000000001e-05 2024-04-26T01:08:32.169 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.169 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.169 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.169 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.344766+0000", 2024-04-26T01:08:32.169 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0063516110000000001 2024-04-26T01:08:32.170 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.170 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.170 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.170 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.344969+0000", 2024-04-26T01:08:32.170 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00020233 2024-04-26T01:08:32.170 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.170 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.170 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.170 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.345881+0000", 2024-04-26T01:08:32.170 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00091244999999999998 2024-04-26T01:08:32.170 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.170 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.170 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.171 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.347458+0000", 2024-04-26T01:08:32.171 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001576945 2024-04-26T01:08:32.171 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.171 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.171 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.171 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.351176+0000", 2024-04-26T01:08:32.171 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0037184449999999999 2024-04-26T01:08:32.171 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.171 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.171 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.171 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.351342+0000", 2024-04-26T01:08:32.171 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016531900000000001 2024-04-26T01:08:32.171 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.171 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.171 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.172 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.172 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.172 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:48 3.ds0 3:b85e6c81:::smithi16428812-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:32.172 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.338562+0000", 2024-04-26T01:08:32.172 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.767638466999998, 2024-04-26T01:08:32.172 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.028604862000000002, 2024-04-26T01:08:32.172 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.172 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.172 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:32.172 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.172 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.172 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.172 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 48 2024-04-26T01:08:32.173 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.173 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.173 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.173 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.173 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338562+0000", 2024-04-26T01:08:32.173 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.173 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.173 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.173 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.173 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338562+0000", 2024-04-26T01:08:32.173 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.173 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.173 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.174 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.174 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338564+0000", 2024-04-26T01:08:32.174 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5319999999999999e-06 2024-04-26T01:08:32.174 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.174 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.174 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.174 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338572+0000", 2024-04-26T01:08:32.174 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1780000000000003e-06 2024-04-26T01:08:32.174 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.174 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.174 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.174 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338587+0000", 2024-04-26T01:08:32.174 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5293000000000001e-05 2024-04-26T01:08:32.174 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.175 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.175 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.175 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338599+0000", 2024-04-26T01:08:32.175 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1968e-05 2024-04-26T01:08:32.175 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.175 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.175 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.175 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346313+0000", 2024-04-26T01:08:32.175 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077139060000000004 2024-04-26T01:08:32.175 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.175 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.175 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:32.175 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346451+0000", 2024-04-26T01:08:32.176 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013819400000000001 2024-04-26T01:08:32.176 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.176 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.176 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.176 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.352103+0000", 2024-04-26T01:08:32.176 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056514249999999999 2024-04-26T01:08:32.176 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.176 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.176 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.176 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.352448+0000", 2024-04-26T01:08:32.176 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034553299999999997 2024-04-26T01:08:32.176 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.176 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.176 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.177 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.367167+0000", 2024-04-26T01:08:32.177 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.014718833000000001 2024-04-26T01:08:32.177 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.177 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.177 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.177 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.177 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.177 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/30 ECSubWrite(tid=2, reqid=client.4521.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:32.177 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.339308+0000", 2024-04-26T01:08:32.177 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.766892323999997, 2024-04-26T01:08:32.177 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0072084560000000002, 2024-04-26T01:08:32.177 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.177 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.177 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:32.177 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.178 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.178 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.178 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339308+0000", 2024-04-26T01:08:32.178 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.178 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.178 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.178 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.178 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339308+0000", 2024-04-26T01:08:32.178 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.178 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.178 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.178 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.178 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339310+0000", 2024-04-26T01:08:32.178 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4959999999999999e-06 2024-04-26T01:08:32.179 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.179 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.179 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.179 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339328+0000", 2024-04-26T01:08:32.179 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8349999999999999e-05 2024-04-26T01:08:32.179 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.179 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.179 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.179 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339614+0000", 2024-04-26T01:08:32.179 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028623 2024-04-26T01:08:32.179 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.179 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.179 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.180 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339629+0000", 2024-04-26T01:08:32.180 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4803999999999999e-05 2024-04-26T01:08:32.180 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.180 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.180 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.180 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.340313+0000", 2024-04-26T01:08:32.180 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00068382200000000005 2024-04-26T01:08:32.180 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.180 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.180 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.180 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.340400+0000", 2024-04-26T01:08:32.180 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.7321e-05 2024-04-26T01:08:32.180 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.180 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.181 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.181 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346462+0000", 2024-04-26T01:08:32.181 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0060617939999999997 2024-04-26T01:08:32.181 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.181 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.181 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.181 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346517+0000", 2024-04-26T01:08:32.181 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4639000000000003e-05 2024-04-26T01:08:32.181 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.181 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.181 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.181 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.181 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.182 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/30 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=38'19, trim_to=0'0, roll_forward_to=37'17))", 2024-04-26T01:08:32.182 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:04.427965+0000", 2024-04-26T01:08:32.182 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 27.678235780000001, 2024-04-26T01:08:32.182 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0080739560000000002, 2024-04-26T01:08:32.182 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.182 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.182 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:32.182 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.182 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.182 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.182 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.427965+0000", 2024-04-26T01:08:32.182 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.182 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.183 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.183 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.183 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.427965+0000", 2024-04-26T01:08:32.183 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.183 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.183 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.183 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.183 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.427966+0000", 2024-04-26T01:08:32.183 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3600000000000003e-07 2024-04-26T01:08:32.183 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.183 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.183 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.183 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.428022+0000", 2024-04-26T01:08:32.184 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6805999999999998e-05 2024-04-26T01:08:32.184 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.184 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.184 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.184 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.428186+0000", 2024-04-26T01:08:32.184 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000163287 2024-04-26T01:08:32.184 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.184 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.184 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.184 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.428212+0000", 2024-04-26T01:08:32.184 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6163e-05 2024-04-26T01:08:32.184 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.184 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.184 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.185 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.430074+0000", 2024-04-26T01:08:32.185 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0018619369999999999 2024-04-26T01:08:32.185 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.185 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.185 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.185 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.430098+0000", 2024-04-26T01:08:32.185 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.393e-05 2024-04-26T01:08:32.185 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.185 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.185 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.185 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.435960+0000", 2024-04-26T01:08:32.185 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0058619930000000002 2024-04-26T01:08:32.186 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.186 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.186 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.186 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.436039+0000", 2024-04-26T01:08:32.186 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9103999999999993e-05 2024-04-26T01:08:32.186 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.186 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.186 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.186 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.186 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.186 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.bs5 55/30 ECSubWrite(tid=252, reqid=client.4521.0:548, at_version=55'28, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:32.186 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:21.086084+0000", 2024-04-26T01:08:32.187 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 11.020116534, 2024-04-26T01:08:32.187 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0057840410000000002, 2024-04-26T01:08:32.187 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.187 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.187 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:32.187 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.187 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.187 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.187 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.086084+0000", 2024-04-26T01:08:32.187 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.187 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.187 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.187 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.187 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.086084+0000", 2024-04-26T01:08:32.188 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.188 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.188 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.188 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.188 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.086086+0000", 2024-04-26T01:08:32.188 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7400000000000001e-06 2024-04-26T01:08:32.188 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.188 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.188 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.188 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.086432+0000", 2024-04-26T01:08:32.188 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034610199999999999 2024-04-26T01:08:32.188 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.188 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.188 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.189 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.086503+0000", 2024-04-26T01:08:32.189 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1166000000000003e-05 2024-04-26T01:08:32.189 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.189 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.189 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.189 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.086518+0000", 2024-04-26T01:08:32.189 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5293000000000001e-05 2024-04-26T01:08:32.189 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.189 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.189 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.189 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.086692+0000", 2024-04-26T01:08:32.189 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000173568 2024-04-26T01:08:32.189 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.190 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.190 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.190 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.086718+0000", 2024-04-26T01:08:32.190 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.622e-05 2024-04-26T01:08:32.190 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.190 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.190 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.190 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.091805+0000", 2024-04-26T01:08:32.190 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0050865800000000003 2024-04-26T01:08:32.190 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.190 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.190 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.190 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.091868+0000", 2024-04-26T01:08:32.191 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3372000000000002e-05 2024-04-26T01:08:32.191 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.191 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.191 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.191 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.191 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.191 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 60/30 ECSubWrite(tid=51, reqid=osd.15.0:51, at_version=60'45, trim_to=0'0, roll_forward_to=59'43))", 2024-04-26T01:08:32.191 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:08:32.191 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 5.9342009779999998, 2024-04-26T01:08:32.191 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.15434805400000001, 2024-04-26T01:08:32.191 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.191 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.191 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:32.191 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.192 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.192 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.192 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:08:32.192 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.192 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.192 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.192 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.192 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:08:32.192 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.192 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.192 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.192 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.192 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:08:32.192 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7800000000000001e-07 2024-04-26T01:08:32.192 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.193 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.193 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.193 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172035+0000", 2024-04-26T01:08:32.193 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.5151999999999998e-05 2024-04-26T01:08:32.193 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.193 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.193 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.193 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324571+0000", 2024-04-26T01:08:32.193 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.152535484 2024-04-26T01:08:32.193 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.193 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.193 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.194 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324589+0000", 2024-04-26T01:08:32.194 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8323999999999999e-05 2024-04-26T01:08:32.194 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.194 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.194 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.194 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324739+0000", 2024-04-26T01:08:32.194 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014940099999999999 2024-04-26T01:08:32.194 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.194 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.194 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.194 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324758+0000", 2024-04-26T01:08:32.194 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9562999999999999e-05 2024-04-26T01:08:32.194 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.194 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.195 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.195 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.326281+0000", 2024-04-26T01:08:32.195 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001523349 2024-04-26T01:08:32.195 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.195 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.195 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.195 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.326348+0000", 2024-04-26T01:08:32.195 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6203000000000003e-05 2024-04-26T01:08:32.195 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.195 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.195 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.195 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.195 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.196 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.bs5 60/30 ECSubWrite(tid=349, reqid=osd.4.0:349, at_version=60'52, trim_to=0'0, roll_forward_to=59'50))", 2024-04-26T01:08:32.196 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:08:32.196 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 5.9286240799999996, 2024-04-26T01:08:32.196 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.007081085, 2024-04-26T01:08:32.196 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.196 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.196 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:32.196 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.196 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.196 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.196 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:08:32.196 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.196 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.197 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.197 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.197 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:08:32.197 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.197 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.197 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.197 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.197 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:08:32.197 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1799999999999999e-07 2024-04-26T01:08:32.197 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.197 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.197 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.197 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177638+0000", 2024-04-26T01:08:32.197 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1736000000000007e-05 2024-04-26T01:08:32.197 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.198 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.198 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.198 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177710+0000", 2024-04-26T01:08:32.198 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1756999999999998e-05 2024-04-26T01:08:32.198 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.198 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.198 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.198 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177722+0000", 2024-04-26T01:08:32.198 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1516e-05 2024-04-26T01:08:32.198 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.198 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.198 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.199 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177868+0000", 2024-04-26T01:08:32.199 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000146112 2024-04-26T01:08:32.199 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.199 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.199 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.199 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177896+0000", 2024-04-26T01:08:32.199 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7943999999999999e-05 2024-04-26T01:08:32.199 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.199 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.199 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.199 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.184603+0000", 2024-04-26T01:08:32.199 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006707691 2024-04-26T01:08:32.199 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.199 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.199 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.200 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.184658+0000", 2024-04-26T01:08:32.200 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4110999999999997e-05 2024-04-26T01:08:32.200 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.200 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.200 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.200 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.200 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.200 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.bs5 60/30 ECSubWrite(tid=351, reqid=osd.4.0:351, at_version=60'54, trim_to=0'0, roll_forward_to=59'50))", 2024-04-26T01:08:32.200 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:26.178472+0000", 2024-04-26T01:08:32.200 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 5.9277284090000002, 2024-04-26T01:08:32.200 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006950861, 2024-04-26T01:08:32.200 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.200 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.200 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178472+0000", 2024-04-26T01:08:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178472+0000", 2024-04-26T01:08:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178474+0000", 2024-04-26T01:08:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2199999999999999e-06 2024-04-26T01:08:32.202 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.202 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.202 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.202 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178532+0000", 2024-04-26T01:08:32.202 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7429000000000002e-05 2024-04-26T01:08:32.202 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.202 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.202 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.202 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178615+0000", 2024-04-26T01:08:32.202 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.3566000000000006e-05 2024-04-26T01:08:32.202 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.202 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.202 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.202 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178639+0000", 2024-04-26T01:08:32.203 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3736000000000001e-05 2024-04-26T01:08:32.203 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.203 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.203 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.203 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178895+0000", 2024-04-26T01:08:32.203 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00025574300000000001 2024-04-26T01:08:32.203 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.203 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.203 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.203 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178920+0000", 2024-04-26T01:08:32.203 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5126999999999998e-05 2024-04-26T01:08:32.203 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.203 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.204 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.204 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.185333+0000", 2024-04-26T01:08:32.204 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0064127389999999998 2024-04-26T01:08:32.204 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.204 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.204 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.204 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.185423+0000", 2024-04-26T01:08:32.204 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0301000000000002e-05 2024-04-26T01:08:32.204 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.204 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.204 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.204 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.204 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.205 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:32.207 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:08:32.207 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:32.201+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:32.218 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-26T01:08:32.307 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:32.304+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:32.349 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:08:32.378 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:32.378 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:32.378 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:08:32.378 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:32.392 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:08:32.438 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:32.439+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:32.509 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:32.505+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:32.513 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:32.513 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:32.516 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:32.517 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:32.517 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2024-04-26T01:08:32.517 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2024-04-26T01:08:32.517 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:32.531 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:08:32.538 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:32.538 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:32.564 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:32.564 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:32.587 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:32.587 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:32.610 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:32.605+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:32.611 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:32.611 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:32.641 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:32.641 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:32.670 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:32.670 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:32.696 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:32.696 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2024-04-26T01:08:32.696 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2024-04-26T01:08:32.696 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2024-04-26T01:08:32.696 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.697 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:28 3.5s0 3:acf4d671:::smithi16428812-10:head [append 0~163840,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:32.697 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321346+0000", 2024-04-26T01:08:32.697 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 33.373369895000003, 2024-04-26T01:08:32.697 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015195917, 2024-04-26T01:08:32.697 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.697 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.697 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.697 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.697 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.697 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.697 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 28 2024-04-26T01:08:32.697 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.697 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.698 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.698 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.698 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321346+0000", 2024-04-26T01:08:32.698 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.698 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.698 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.698 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.698 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321346+0000", 2024-04-26T01:08:32.698 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.698 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.698 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.698 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.698 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321349+0000", 2024-04-26T01:08:32.699 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0879999999999999e-06 2024-04-26T01:08:32.699 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.699 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.699 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.699 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321587+0000", 2024-04-26T01:08:32.699 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000237698 2024-04-26T01:08:32.699 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.699 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.699 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.699 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321610+0000", 2024-04-26T01:08:32.699 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2597000000000001e-05 2024-04-26T01:08:32.699 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.699 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.699 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.700 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321624+0000", 2024-04-26T01:08:32.700 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4365999999999999e-05 2024-04-26T01:08:32.700 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.700 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.700 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.700 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321789+0000", 2024-04-26T01:08:32.700 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016444499999999999 2024-04-26T01:08:32.700 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.700 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.700 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.700 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322181+0000", 2024-04-26T01:08:32.700 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00039229 2024-04-26T01:08:32.700 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.700 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.700 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.701 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324455+0000", 2024-04-26T01:08:32.701 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0022738979999999999 2024-04-26T01:08:32.701 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.701 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.701 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.701 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.334069+0000", 2024-04-26T01:08:32.701 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.009614793 2024-04-26T01:08:32.701 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.701 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.701 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.701 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336369+0000", 2024-04-26T01:08:32.701 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0022993689999999999 2024-04-26T01:08:32.701 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.701 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.702 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.702 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336542+0000", 2024-04-26T01:08:32.702 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017337299999999999 2024-04-26T01:08:32.702 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.702 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.702 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.702 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.702 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.702 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:29 3.5s0 3:acf4d671:::smithi16428812-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:32.702 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321678+0000", 2024-04-26T01:08:32.702 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 33.373037848999999, 2024-04-26T01:08:32.702 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.016320530999999999, 2024-04-26T01:08:32.702 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.702 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.703 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.703 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.703 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.703 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.703 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 29 2024-04-26T01:08:32.703 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.703 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.703 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.703 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.703 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321678+0000", 2024-04-26T01:08:32.703 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.703 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.703 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.703 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.704 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321678+0000", 2024-04-26T01:08:32.704 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.704 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.704 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.704 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.704 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321680+0000", 2024-04-26T01:08:32.704 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1680000000000001e-06 2024-04-26T01:08:32.704 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.704 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.704 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.704 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321686+0000", 2024-04-26T01:08:32.704 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9539999999999998e-06 2024-04-26T01:08:32.704 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.704 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.704 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.705 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321702+0000", 2024-04-26T01:08:32.705 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5241e-05 2024-04-26T01:08:32.705 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.705 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.705 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.705 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321716+0000", 2024-04-26T01:08:32.705 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4384e-05 2024-04-26T01:08:32.705 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.705 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.705 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.705 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329428+0000", 2024-04-26T01:08:32.705 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077114369999999998 2024-04-26T01:08:32.705 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.705 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.705 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.706 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329669+0000", 2024-04-26T01:08:32.706 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000241561 2024-04-26T01:08:32.706 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.706 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.706 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.706 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.331103+0000", 2024-04-26T01:08:32.706 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001433871 2024-04-26T01:08:32.706 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.706 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.706 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.706 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336734+0000", 2024-04-26T01:08:32.706 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.005631306 2024-04-26T01:08:32.706 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.706 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.706 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.707 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337654+0000", 2024-04-26T01:08:32.707 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000919716 2024-04-26T01:08:32.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.707 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.707 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337999+0000", 2024-04-26T01:08:32.707 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034489299999999998 2024-04-26T01:08:32.707 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.707 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.707 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.707 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:30 3.5s0 3:acf4d671:::smithi16428812-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:32.707 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.322117+0000", 2024-04-26T01:08:32.708 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 33.372598793999998, 2024-04-26T01:08:32.708 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.021239319999999999, 2024-04-26T01:08:32.708 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.708 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.708 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:32.708 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.708 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.708 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.708 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 30 2024-04-26T01:08:32.708 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.708 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.708 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.708 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.708 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322117+0000", 2024-04-26T01:08:32.708 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.709 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.709 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322117+0000", 2024-04-26T01:08:32.709 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.709 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.709 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322119+0000", 2024-04-26T01:08:32.709 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3740000000000001e-06 2024-04-26T01:08:32.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.709 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.709 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322127+0000", 2024-04-26T01:08:32.709 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.994e-06 2024-04-26T01:08:32.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.710 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.710 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.710 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322146+0000", 2024-04-26T01:08:32.710 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9205999999999999e-05 2024-04-26T01:08:32.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.710 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.710 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.710 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322159+0000", 2024-04-26T01:08:32.710 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3377000000000001e-05 2024-04-26T01:08:32.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.710 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.710 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.711 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.334766+0000", 2024-04-26T01:08:32.711 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012606732000000001 2024-04-26T01:08:32.711 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.711 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.711 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:32.711 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335000+0000", 2024-04-26T01:08:32.711 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00023435799999999999 2024-04-26T01:08:32.711 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.711 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.711 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.711 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338932+0000", 2024-04-26T01:08:32.711 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0039317500000000003 2024-04-26T01:08:32.711 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.711 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.711 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.712 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339070+0000", 2024-04-26T01:08:32.712 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000137315 2024-04-26T01:08:32.712 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.712 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.712 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.712 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.343357+0000", 2024-04-26T01:08:32.712 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042872140000000001 2024-04-26T01:08:32.712 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.712 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.712 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.712 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.712 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.712 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:84 3.5s0 3:a8ed6f44:::smithi16428812-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:32.712 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.390471+0000", 2024-04-26T01:08:32.713 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 33.304244775999997, 2024-04-26T01:08:32.713 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012687102, 2024-04-26T01:08:32.713 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.713 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.713 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:32.713 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.713 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.713 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.713 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 84 2024-04-26T01:08:32.713 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.713 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.713 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.713 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.713 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.390471+0000", 2024-04-26T01:08:32.713 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.714 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.714 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.714 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.714 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.390471+0000", 2024-04-26T01:08:32.714 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.714 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.714 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.714 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.714 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.390472+0000", 2024-04-26T01:08:32.714 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3200000000000004e-07 2024-04-26T01:08:32.715 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.715 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.715 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.715 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.390475+0000", 2024-04-26T01:08:32.715 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6740000000000001e-06 2024-04-26T01:08:32.715 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.715 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.715 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.715 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.390482+0000", 2024-04-26T01:08:32.715 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6260000000000003e-06 2024-04-26T01:08:32.715 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.715 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.715 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.716 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.390488+0000", 2024-04-26T01:08:32.716 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7479999999999999e-06 2024-04-26T01:08:32.716 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.716 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.716 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.716 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394836+0000", 2024-04-26T01:08:32.716 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0043482 2024-04-26T01:08:32.716 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.716 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.394948+0000", 2024-04-26T01:08:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011179399999999999 2024-04-26T01:08:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.400983+0000", 2024-04-26T01:08:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0060351190000000002 2024-04-26T01:08:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.401097+0000", 2024-04-26T01:08:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000113773 2024-04-26T01:08:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.403159+0000", 2024-04-26T01:08:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0020614359999999998 2024-04-26T01:08:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:128 3.5s0 3:a661dd24:::smithi16428812-43:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 33.238230670999997, 2024-04-26T01:08:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.083673129999999998, 2024-04-26T01:08:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 128 2024-04-26T01:08:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1299999999999999e-07 2024-04-26T01:08:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456490+0000", 2024-04-26T01:08:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4029999999999999e-06 2024-04-26T01:08:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456501+0000", 2024-04-26T01:08:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1124999999999999e-05 2024-04-26T01:08:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456507+0000", 2024-04-26T01:08:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3180000000000003e-06 2024-04-26T01:08:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.458775+0000", 2024-04-26T01:08:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002268061 2024-04-26T01:08:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.458948+0000", 2024-04-26T01:08:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017263900000000001 2024-04-26T01:08:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.459498+0000", 2024-04-26T01:08:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00055037900000000004 2024-04-26T01:08:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460166+0000", 2024-04-26T01:08:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00066757899999999996 2024-04-26T01:08:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540012+0000", 2024-04-26T01:08:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.079846146000000007 2024-04-26T01:08:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540159+0000", 2024-04-26T01:08:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014676700000000001 2024-04-26T01:08:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:129 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 33.238201699999998, 2024-04-26T01:08:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.09570555, 2024-04-26T01:08:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 129 2024-04-26T01:08:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0699999999999997e-07 2024-04-26T01:08:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456518+0000", 2024-04-26T01:08:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.898e-06 2024-04-26T01:08:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456532+0000", 2024-04-26T01:08:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3597e-05 2024-04-26T01:08:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456537+0000", 2024-04-26T01:08:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.242e-06 2024-04-26T01:08:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.459839+0000", 2024-04-26T01:08:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033019830000000001 2024-04-26T01:08:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.459972+0000", 2024-04-26T01:08:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000133199 2024-04-26T01:08:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540208+0000", 2024-04-26T01:08:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.080236319 2024-04-26T01:08:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540275+0000", 2024-04-26T01:08:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6486e-05 2024-04-26T01:08:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.552220+0000", 2024-04-26T01:08:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011945318999999999 2024-04-26T01:08:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.731 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.731 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.731 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.731 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 33/30 ECSubWriteReply(tid=9, last_complete=33'6, committed=1, applied=1))", 2024-04-26T01:08:32.731 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:32.731 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 33.234420626000002, 2024-04-26T01:08:32.731 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.079879818000000005, 2024-04-26T01:08:32.731 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.731 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.731 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:32.731 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.731 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.731 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.731 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:32.731 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6599999999999996e-07 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460301+0000", 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4689999999999999e-06 2024-04-26T01:08:32.732 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.539830+0000", 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.079528926999999999 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.539834+0000", 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5430000000000004e-06 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.539887+0000", 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2998999999999999e-05 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.733 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.734 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.734 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540175+0000", 2024-04-26T01:08:32.734 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028821400000000002 2024-04-26T01:08:32.734 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.734 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.734 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.734 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.734 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.734 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:182 3.5s0 3:a8ed6f44:::smithi16428812-28:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:32.734 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.535158+0000", 2024-04-26T01:08:32.734 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 33.159557972999998, 2024-04-26T01:08:32.734 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015264214, 2024-04-26T01:08:32.734 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.734 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.735 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.735 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.735 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.735 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.735 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 182 2024-04-26T01:08:32.735 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.735 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.735 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.735 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.735 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535158+0000", 2024-04-26T01:08:32.735 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.735 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.735 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.735 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.736 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535158+0000", 2024-04-26T01:08:32.736 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.736 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.736 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.736 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.736 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535159+0000", 2024-04-26T01:08:32.736 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.71e-07 2024-04-26T01:08:32.736 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.736 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.736 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.736 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535163+0000", 2024-04-26T01:08:32.736 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1540000000000004e-06 2024-04-26T01:08:32.736 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.736 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.736 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535173+0000", 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.5230000000000001e-06 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535180+0000", 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1629999999999999e-06 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.536667+0000", 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014868850000000001 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.737 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.536801+0000", 2024-04-26T01:08:32.738 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013424500000000001 2024-04-26T01:08:32.738 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.738 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.738 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.738 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.537375+0000", 2024-04-26T01:08:32.738 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00057366299999999997 2024-04-26T01:08:32.738 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.738 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.738 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.738 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.538137+0000", 2024-04-26T01:08:32.738 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00076241899999999997 2024-04-26T01:08:32.738 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.738 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.738 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.738 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.550301+0000", 2024-04-26T01:08:32.739 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012163386 2024-04-26T01:08:32.739 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.739 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.739 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.739 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.550422+0000", 2024-04-26T01:08:32.739 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012190500000000001 2024-04-26T01:08:32.739 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.739 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.739 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.739 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.739 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.739 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:183 3.5s0 3:a8ed6f44:::smithi16428812-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:32.739 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:32.739 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 33.159522637999999, 2024-04-26T01:08:32.740 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.017578818999999999, 2024-04-26T01:08:32.740 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.740 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.740 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:32.740 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.740 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.740 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.740 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 183 2024-04-26T01:08:32.740 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.740 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.740 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.740 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.740 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:32.740 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.740 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.741 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.741 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.741 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:32.741 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.741 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.741 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.741 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.741 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:32.741 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9500000000000002e-07 2024-04-26T01:08:32.741 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.741 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.741 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.741 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535198+0000", 2024-04-26T01:08:32.741 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6430000000000001e-06 2024-04-26T01:08:32.742 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.742 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.742 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.742 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535207+0000", 2024-04-26T01:08:32.742 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9730000000000003e-06 2024-04-26T01:08:32.742 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.742 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.742 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.742 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535215+0000", 2024-04-26T01:08:32.742 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1540000000000002e-06 2024-04-26T01:08:32.742 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.742 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.742 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.742 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.537794+0000", 2024-04-26T01:08:32.742 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0025788740000000001 2024-04-26T01:08:32.743 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.743 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.743 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:32.743 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.537868+0000", 2024-04-26T01:08:32.743 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4028000000000003e-05 2024-04-26T01:08:32.743 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.743 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.743 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.743 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.550967+0000", 2024-04-26T01:08:32.743 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013099006999999999 2024-04-26T01:08:32.743 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.743 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.743 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.743 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551052+0000", 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4667999999999996e-05 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.552772+0000", 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0017207769999999999 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 33/30 ECSubWriteReply(tid=14, last_complete=33'9, committed=1, applied=1))", 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.551733+0000", 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 33.142983176999998, 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.40949085299999999, 2024-04-26T01:08:32.744 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.745 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.745 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:32.745 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.745 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.745 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.745 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551733+0000", 2024-04-26T01:08:32.745 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.745 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.745 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.745 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.745 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551733+0000", 2024-04-26T01:08:32.745 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.745 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.745 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.746 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.746 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551734+0000", 2024-04-26T01:08:32.746 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7199999999999998e-07 2024-04-26T01:08:32.746 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.746 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.746 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.746 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551739+0000", 2024-04-26T01:08:32.746 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9520000000000002e-06 2024-04-26T01:08:32.746 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.746 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.746 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.746 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.960976+0000", 2024-04-26T01:08:32.746 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.40923691400000001 2024-04-26T01:08:32.746 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.746 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.960989+0000", 2024-04-26T01:08:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3536000000000001e-05 2024-04-26T01:08:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.961157+0000", 2024-04-26T01:08:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016794600000000001 2024-04-26T01:08:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.961224+0000", 2024-04-26T01:08:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6733e-05 2024-04-26T01:08:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.748 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.748 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.748 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.748 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:251 3.5s0 3:acf4d671:::smithi16428812-10:head [append 0~0] snapc 4=[4,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:32.748 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:03.428674+0000", 2024-04-26T01:08:32.748 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 29.266041739999999, 2024-04-26T01:08:32.748 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.48188827299999998, 2024-04-26T01:08:32.748 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.748 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.748 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.748 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.748 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.748 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.748 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 251 2024-04-26T01:08:32.748 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428674+0000", 2024-04-26T01:08:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428674+0000", 2024-04-26T01:08:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428676+0000", 2024-04-26T01:08:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6559999999999999e-06 2024-04-26T01:08:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428685+0000", 2024-04-26T01:08:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0869999999999996e-06 2024-04-26T01:08:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428707+0000", 2024-04-26T01:08:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1365999999999998e-05 2024-04-26T01:08:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428723+0000", 2024-04-26T01:08:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6563e-05 2024-04-26T01:08:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.434334+0000", 2024-04-26T01:08:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056104170000000004 2024-04-26T01:08:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.434677+0000", 2024-04-26T01:08:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000343937 2024-04-26T01:08:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.436169+0000", 2024-04-26T01:08:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014915939999999999 2024-04-26T01:08:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.438266+0000", 2024-04-26T01:08:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002096746 2024-04-26T01:08:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.910445+0000", 2024-04-26T01:08:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.47217926300000002 2024-04-26T01:08:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.910563+0000", 2024-04-26T01:08:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000117644 2024-04-26T01:08:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:412 3.5s0 3:a661dd24:::smithi16428812-43:head [append 0~688128] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-26T01:08:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.437736+0000", 2024-04-26T01:08:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 22.256980292000001, 2024-04-26T01:08:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015629959999999998, 2024-04-26T01:08:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 412 2024-04-26T01:08:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.437736+0000", 2024-04-26T01:08:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.437736+0000", 2024-04-26T01:08:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.437739+0000", 2024-04-26T01:08:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1530000000000001e-06 2024-04-26T01:08:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.438458+0000", 2024-04-26T01:08:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00071916899999999997 2024-04-26T01:08:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.438496+0000", 2024-04-26T01:08:32.756 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8022999999999998e-05 2024-04-26T01:08:32.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.756 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.756 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.756 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.438516+0000", 2024-04-26T01:08:32.756 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9936e-05 2024-04-26T01:08:32.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.756 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.756 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.756 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.438651+0000", 2024-04-26T01:08:32.756 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013456899999999999 2024-04-26T01:08:32.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.756 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.756 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2024-04-26T01:08:32.757 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.438661+0000", 2024-04-26T01:08:32.757 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0363000000000001e-05 2024-04-26T01:08:32.757 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.757 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.757 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.442159+0000", 2024-04-26T01:08:32.757 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0034977509999999999 2024-04-26T01:08:32.757 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.757 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.757 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.442304+0000", 2024-04-26T01:08:32.757 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014504599999999999 2024-04-26T01:08:32.757 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.757 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.758 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.444152+0000", 2024-04-26T01:08:32.758 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0018484669999999999 2024-04-26T01:08:32.758 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.758 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.758 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.758 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.452613+0000", 2024-04-26T01:08:32.758 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0084610940000000006 2024-04-26T01:08:32.758 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.758 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.758 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.758 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.453249+0000", 2024-04-26T01:08:32.758 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000635989 2024-04-26T01:08:32.758 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.758 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.759 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.759 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.453366+0000", 2024-04-26T01:08:32.759 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0001164 2024-04-26T01:08:32.759 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.759 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.759 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.759 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.759 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.759 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:413 3.5s0 3:a661dd24:::smithi16428812-43:head [append 0~540672] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-26T01:08:32.759 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.442205+0000", 2024-04-26T01:08:32.759 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 22.252510852, 2024-04-26T01:08:32.759 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011472952, 2024-04-26T01:08:32.759 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.759 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.759 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.760 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.760 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.760 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.760 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 413 2024-04-26T01:08:32.760 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.760 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.760 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.760 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.760 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.442205+0000", 2024-04-26T01:08:32.760 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.760 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.760 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.760 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.760 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.442205+0000", 2024-04-26T01:08:32.760 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.761 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.761 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.761 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.761 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.442209+0000", 2024-04-26T01:08:32.761 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7730000000000001e-06 2024-04-26T01:08:32.761 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.761 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.761 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.761 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.442992+0000", 2024-04-26T01:08:32.761 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00078293099999999995 2024-04-26T01:08:32.761 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.761 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.761 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.761 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443016+0000", 2024-04-26T01:08:32.761 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.376e-05 2024-04-26T01:08:32.762 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.762 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.762 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443033+0000", 2024-04-26T01:08:32.762 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7479e-05 2024-04-26T01:08:32.762 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.762 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.762 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.446209+0000", 2024-04-26T01:08:32.762 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0031761599999999999 2024-04-26T01:08:32.762 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.762 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.762 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.446534+0000", 2024-04-26T01:08:32.763 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00032445500000000002 2024-04-26T01:08:32.763 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.763 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.763 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.763 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.448373+0000", 2024-04-26T01:08:32.763 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0018392580000000001 2024-04-26T01:08:32.763 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.763 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.763 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.763 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.452615+0000", 2024-04-26T01:08:32.763 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042421230000000004 2024-04-26T01:08:32.763 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.763 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.763 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.764 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.453618+0000", 2024-04-26T01:08:32.764 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010031319999999999 2024-04-26T01:08:32.764 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.764 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.764 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.764 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.453678+0000", 2024-04-26T01:08:32.764 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9880999999999998e-05 2024-04-26T01:08:32.764 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.764 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.764 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.764 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.764 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.764 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:414 3.5s0 3:a661dd24:::smithi16428812-43:head [append 0~229376] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-26T01:08:32.764 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.443547+0000", 2024-04-26T01:08:32.765 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 22.251169033, 2024-04-26T01:08:32.765 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011928169000000001, 2024-04-26T01:08:32.765 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.765 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.765 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.765 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.765 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.765 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.765 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 414 2024-04-26T01:08:32.765 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.765 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.765 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.765 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.765 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443547+0000", 2024-04-26T01:08:32.765 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.766 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.766 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.766 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.766 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443547+0000", 2024-04-26T01:08:32.766 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.766 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.766 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.766 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.766 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443550+0000", 2024-04-26T01:08:32.766 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0060000000000001e-06 2024-04-26T01:08:32.766 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.766 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.766 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.767 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443876+0000", 2024-04-26T01:08:32.767 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00032578199999999999 2024-04-26T01:08:32.767 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.767 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.767 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.767 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443897+0000", 2024-04-26T01:08:32.767 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1433e-05 2024-04-26T01:08:32.767 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.767 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.767 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.767 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443910+0000", 2024-04-26T01:08:32.767 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2969000000000001e-05 2024-04-26T01:08:32.767 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.767 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.767 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.768 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.449751+0000", 2024-04-26T01:08:32.768 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0058404659999999999 2024-04-26T01:08:32.768 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.768 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.768 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.768 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.449999+0000", 2024-04-26T01:08:32.768 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024795900000000002 2024-04-26T01:08:32.768 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.768 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.768 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.768 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.451445+0000", 2024-04-26T01:08:32.768 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001446391 2024-04-26T01:08:32.769 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.769 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.769 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.769 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.453555+0000", 2024-04-26T01:08:32.769 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021102809999999999 2024-04-26T01:08:32.769 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.769 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.769 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.769 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.455407+0000", 2024-04-26T01:08:32.769 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0018518059999999999 2024-04-26T01:08:32.769 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.769 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.769 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.770 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.455475+0000", 2024-04-26T01:08:32.770 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8076000000000003e-05 2024-04-26T01:08:32.770 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.770 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.770 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.770 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.770 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.770 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:415 3.5s0 3:a661dd24:::smithi16428812-43:head [setxattr _header (56) in=7b] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-26T01:08:32.770 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.443941+0000", 2024-04-26T01:08:32.770 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 22.250775693000001, 2024-04-26T01:08:32.770 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.016144703999999999, 2024-04-26T01:08:32.770 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.770 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.770 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.771 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.771 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.771 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.771 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 415 2024-04-26T01:08:32.771 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.771 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.771 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.771 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.771 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443941+0000", 2024-04-26T01:08:32.771 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.771 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.771 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.771 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.771 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443941+0000", 2024-04-26T01:08:32.772 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.772 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.772 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.772 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.772 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443942+0000", 2024-04-26T01:08:32.772 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1999999999999999e-06 2024-04-26T01:08:32.772 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.772 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.772 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.772 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443950+0000", 2024-04-26T01:08:32.772 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.0639999999999994e-06 2024-04-26T01:08:32.772 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.772 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.772 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.772 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443961+0000", 2024-04-26T01:08:32.773 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0798999999999999e-05 2024-04-26T01:08:32.773 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.773 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.773 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.773 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443968+0000", 2024-04-26T01:08:32.773 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6699999999999994e-06 2024-04-26T01:08:32.773 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.773 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.773 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.773 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.453783+0000", 2024-04-26T01:08:32.773 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0098143120000000004 2024-04-26T01:08:32.773 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.773 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.773 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.773 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.453843+0000", 2024-04-26T01:08:32.774 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0201000000000002e-05 2024-04-26T01:08:32.774 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.774 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.774 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.774 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.454111+0000", 2024-04-26T01:08:32.774 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00026851700000000002 2024-04-26T01:08:32.774 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.774 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.774 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.774 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.455183+0000", 2024-04-26T01:08:32.774 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010722030000000001 2024-04-26T01:08:32.774 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.774 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.774 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.774 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.459986+0000", 2024-04-26T01:08:32.775 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0048022350000000002 2024-04-26T01:08:32.775 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.775 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.775 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.775 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.460085+0000", 2024-04-26T01:08:32.775 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9502999999999999e-05 2024-04-26T01:08:32.775 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.775 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.775 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.775 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.775 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.775 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:416 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-26T01:08:32.775 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.443980+0000", 2024-04-26T01:08:32.775 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 22.25073635, 2024-04-26T01:08:32.776 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.018880322000000001, 2024-04-26T01:08:32.776 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.776 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.776 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:32.776 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.776 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.776 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.776 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 416 2024-04-26T01:08:32.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.776 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.776 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.776 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.776 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443980+0000", 2024-04-26T01:08:32.776 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.777 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.777 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443980+0000", 2024-04-26T01:08:32.777 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.777 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.777 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.777 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443981+0000", 2024-04-26T01:08:32.777 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7000000000000003e-07 2024-04-26T01:08:32.777 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.777 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.777 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443984+0000", 2024-04-26T01:08:32.777 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.36e-06 2024-04-26T01:08:32.778 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.778 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.778 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.778 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443994+0000", 2024-04-26T01:08:32.778 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0285999999999999e-05 2024-04-26T01:08:32.778 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.778 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.778 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.778 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.444002+0000", 2024-04-26T01:08:32.778 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4970000000000004e-06 2024-04-26T01:08:32.778 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.778 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.778 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.778 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.454900+0000", 2024-04-26T01:08:32.779 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010897784000000001 2024-04-26T01:08:32.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.779 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:32.779 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.455102+0000", 2024-04-26T01:08:32.779 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00020183000000000001 2024-04-26T01:08:32.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.779 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.779 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.460460+0000", 2024-04-26T01:08:32.779 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0053580140000000004 2024-04-26T01:08:32.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.779 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.779 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.460760+0000", 2024-04-26T01:08:32.780 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00030007900000000002 2024-04-26T01:08:32.780 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.780 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.780 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.780 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.462860+0000", 2024-04-26T01:08:32.780 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021005020000000002 2024-04-26T01:08:32.780 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.780 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.780 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.780 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.780 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.780 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.3s3 44/30 ECSubWrite(tid=198, reqid=client.4521.0:420, at_version=44'13, trim_to=0'0, roll_forward_to=44'11))", 2024-04-26T01:08:32.780 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.466337+0000", 2024-04-26T01:08:32.781 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 22.228379593, 2024-04-26T01:08:32.781 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.57864738100000002, 2024-04-26T01:08:32.781 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.781 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.781 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:32.781 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.781 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.781 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.781 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466337+0000", 2024-04-26T01:08:32.781 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.781 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.781 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.781 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466337+0000", 2024-04-26T01:08:32.782 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.782 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.782 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.782 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466338+0000", 2024-04-26T01:08:32.782 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4479999999999999e-06 2024-04-26T01:08:32.782 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.782 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.782 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466374+0000", 2024-04-26T01:08:32.782 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6137999999999999e-05 2024-04-26T01:08:32.782 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.782 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.783 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.043926+0000", 2024-04-26T01:08:32.783 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.57755188800000001 2024-04-26T01:08:32.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.783 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.783 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.043944+0000", 2024-04-26T01:08:32.783 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7745999999999999e-05 2024-04-26T01:08:32.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.783 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.783 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044049+0000", 2024-04-26T01:08:32.783 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010500700000000001 2024-04-26T01:08:32.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.784 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.784 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044068+0000", 2024-04-26T01:08:32.784 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8984999999999999e-05 2024-04-26T01:08:32.784 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.784 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.784 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044949+0000", 2024-04-26T01:08:32.784 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00088138400000000003 2024-04-26T01:08:32.784 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.784 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.784 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044984+0000", 2024-04-26T01:08:32.784 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4785e-05 2024-04-26T01:08:32.784 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.785 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.785 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.785 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.785 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.785 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s4 45/44 ECSubWrite(tid=204, reqid=client.4521.0:364, at_version=45'18, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:32.785 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:11.085534+0000", 2024-04-26T01:08:32.785 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 21.609181795000001, 2024-04-26T01:08:32.785 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011269358, 2024-04-26T01:08:32.785 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.785 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.785 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:32.785 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.785 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.785 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.785 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.085534+0000", 2024-04-26T01:08:32.786 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.786 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.786 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.786 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.786 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.085534+0000", 2024-04-26T01:08:32.786 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.786 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.786 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.786 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.786 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.085536+0000", 2024-04-26T01:08:32.786 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.15e-06 2024-04-26T01:08:32.786 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.786 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.786 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.787 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.085702+0000", 2024-04-26T01:08:32.787 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016618399999999999 2024-04-26T01:08:32.787 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.787 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.787 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.787 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.094903+0000", 2024-04-26T01:08:32.787 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0092013500000000005 2024-04-26T01:08:32.787 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.787 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.787 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.787 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.094926+0000", 2024-04-26T01:08:32.787 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3133e-05 2024-04-26T01:08:32.787 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.787 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.787 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.788 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.095048+0000", 2024-04-26T01:08:32.788 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000121722 2024-04-26T01:08:32.788 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.788 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.788 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.788 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.095057+0000", 2024-04-26T01:08:32.788 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3319999999999998e-06 2024-04-26T01:08:32.788 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.788 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.788 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.788 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.096713+0000", 2024-04-26T01:08:32.788 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0016560310000000001 2024-04-26T01:08:32.788 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.788 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.789 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.789 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.096804+0000", 2024-04-26T01:08:32.789 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0456000000000003e-05 2024-04-26T01:08:32.789 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.789 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.789 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.789 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.789 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.789 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:554 3.5s0 3:a661dd24:::smithi16428812-43:head [copy-from ver 26] snapc e=[e,d,b,7] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-04-26T01:08:32.789 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:21.094110+0000", 2024-04-26T01:08:32.789 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 11.600605804000001, 2024-04-26T01:08:32.789 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.050017671, 2024-04-26T01:08:32.789 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.789 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.790 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:32.790 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.790 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.790 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.790 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 554 2024-04-26T01:08:32.790 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.790 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.790 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.790 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.790 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094110+0000", 2024-04-26T01:08:32.790 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.790 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.790 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.790 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.791 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094110+0000", 2024-04-26T01:08:32.791 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.791 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.791 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.791 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.791 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094114+0000", 2024-04-26T01:08:32.791 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.89e-06 2024-04-26T01:08:32.791 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.791 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.791 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.791 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094134+0000", 2024-04-26T01:08:32.791 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0007e-05 2024-04-26T01:08:32.791 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.791 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.792 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.792 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094181+0000", 2024-04-26T01:08:32.792 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6731999999999999e-05 2024-04-26T01:08:32.792 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.792 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.792 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.792 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094205+0000", 2024-04-26T01:08:32.792 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4326999999999999e-05 2024-04-26T01:08:32.792 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.792 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.792 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.792 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094455+0000", 2024-04-26T01:08:32.792 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024929999999999999 2024-04-26T01:08:32.792 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.792 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.793 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2024-04-26T01:08:32.793 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094477+0000", 2024-04-26T01:08:32.793 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2646e-05 2024-04-26T01:08:32.793 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.793 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.793 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.793 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105141+0000", 2024-04-26T01:08:32.793 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010664167 2024-04-26T01:08:32.793 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.793 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.793 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.793 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105392+0000", 2024-04-26T01:08:32.793 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00025070700000000002 2024-04-26T01:08:32.793 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.793 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.794 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:32.794 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.135272+0000", 2024-04-26T01:08:32.794 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.029880263000000001 2024-04-26T01:08:32.794 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.794 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.794 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:32.794 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.142569+0000", 2024-04-26T01:08:32.794 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0072964270000000003 2024-04-26T01:08:32.794 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.794 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.794 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:32.794 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.142755+0000", 2024-04-26T01:08:32.794 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00018603399999999999 2024-04-26T01:08:32.794 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.794 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.795 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.795 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.144128+0000", 2024-04-26T01:08:32.795 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0013731710000000001 2024-04-26T01:08:32.795 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.795 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.795 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.795 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.795 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.795 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:555 3.5s0 3:a661dd24:::smithi16428812-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e55)", 2024-04-26T01:08:32.795 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:21.094258+0000", 2024-04-26T01:08:32.795 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 11.600458638999999, 2024-04-26T01:08:32.795 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.045831917999999999, 2024-04-26T01:08:32.795 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:32.795 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:32.796 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:32.796 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:32.796 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:32.796 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:32.796 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 555 2024-04-26T01:08:32.796 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.796 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:32.796 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.796 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:32.796 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094258+0000", 2024-04-26T01:08:32.796 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.796 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.796 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.796 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:32.796 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094258+0000", 2024-04-26T01:08:32.797 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:32.797 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.797 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.797 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:32.797 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094259+0000", 2024-04-26T01:08:32.797 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7370000000000001e-06 2024-04-26T01:08:32.797 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.797 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.797 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:32.797 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094268+0000", 2024-04-26T01:08:32.797 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8359999999999998e-06 2024-04-26T01:08:32.797 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.797 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.797 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094289+0000", 2024-04-26T01:08:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0937e-05 2024-04-26T01:08:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094299+0000", 2024-04-26T01:08:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.029e-05 2024-04-26T01:08:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094609+0000", 2024-04-26T01:08:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00031005900000000001 2024-04-26T01:08:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094619+0000", 2024-04-26T01:08:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9799999999999993e-06 2024-04-26T01:08:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105868+0000", 2024-04-26T01:08:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011248555 2024-04-26T01:08:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2024-04-26T01:08:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105946+0000", 2024-04-26T01:08:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8005999999999998e-05 2024-04-26T01:08:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:32.800 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.139672+0000", 2024-04-26T01:08:32.800 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.033725805999999997 2024-04-26T01:08:32.800 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.800 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.800 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:32.800 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.139869+0000", 2024-04-26T01:08:32.800 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019683 2024-04-26T01:08:32.800 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:32.800 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:32.800 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:32.800 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.140089+0000", 2024-04-26T01:08:32.800 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022088199999999999 2024-04-26T01:08:32.800 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.800 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.800 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.801 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:32.801 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:32.801 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:32.802 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:32.802 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:32.803 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:32.803 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:32.803 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:08:32.804 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:32.804 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:32.805 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:32.805 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:32.805 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:32.797+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:32.828 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:08:32.828 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:08:32.863 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:32.863 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:32.898 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:32.898 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:32.903 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:32.903+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:32.931 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:32.932 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:32.932 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:32.933 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:32.933 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:32.928+0000 7fdd394a9640 0 -- 172.21.15.8:0/3743619131 >> [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] conn(0x7fdd34223f40 msgr2=0x7fdd34226330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:08:32.946 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:08:33.004 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:33.000+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:33.043 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.2 to 1 (10000) 2024-04-26T01:08:33.043 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-04-26T01:08:33.047 INFO:tasks.rados.rados.0.smithi164.stdout:275: done (2 left) 2024-04-26T01:08:33.048 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 38 current snap is 17 2024-04-26T01:08:33.048 INFO:tasks.rados.rados.0.smithi164.stdout:276: seq_num 124 ranges {0=540672,540672=720896,1261568=786432,2048000=49152} 2024-04-26T01:08:33.052 INFO:tasks.rados.rados.0.smithi164.stdout:276: writing smithi16428812-38 from 0 to 540672 tid 1 2024-04-26T01:08:33.057 INFO:tasks.rados.rados.0.smithi164.stdout:276: writing smithi16428812-38 from 540672 to 1261568 tid 2 2024-04-26T01:08:33.069 INFO:tasks.rados.rados.0.smithi164.stdout:276: writing smithi16428812-38 from 1261568 to 2048000 tid 3 2024-04-26T01:08:33.071 INFO:tasks.rados.rados.0.smithi164.stdout:276: writing smithi16428812-38 from 2048000 to 2097152 tid 4 2024-04-26T01:08:33.071 INFO:tasks.rados.rados.0.smithi164.stdout:277: copy_from oid 7 from oid 15 current snap is 17 2024-04-26T01:08:33.071 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 30 v 70 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-26T01:08:33.071 INFO:tasks.rados.rados.0.smithi164.stdout:271: finishing write tid 5 to smithi16428812-34 2024-04-26T01:08:33.071 INFO:tasks.rados.rados.0.smithi164.stdout:271: finishing write tid 6 to smithi16428812-34 2024-04-26T01:08:33.071 INFO:tasks.rados.rados.0.smithi164.stdout:271: finishing write tid 7 to smithi16428812-34 2024-04-26T01:08:33.071 INFO:tasks.rados.rados.0.smithi164.stdout:271: oid 34 updating version 0 to 80 2024-04-26T01:08:33.072 INFO:tasks.rados.rados.0.smithi164.stdout:271: oid 34 updating version 80 to 82 2024-04-26T01:08:33.072 INFO:tasks.rados.rados.0.smithi164.stdout:271: oid 34 updating version 82 to 83 2024-04-26T01:08:33.072 INFO:tasks.rados.rados.0.smithi164.stdout:271: oid 34 updating version 83 to 84 2024-04-26T01:08:33.072 INFO:tasks.rados.rados.0.smithi164.stdout:271: oid 34 version 84 is already newer than 79 2024-04-26T01:08:33.072 INFO:tasks.rados.rados.0.smithi164.stdout:271: oid 34 version 84 is already newer than 81 2024-04-26T01:08:33.072 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 34 v 84 (ObjNum 122 snap 17 seq_num 122) dirty exists 2024-04-26T01:08:33.072 INFO:tasks.rados.rados.0.smithi164.stdout:271: left oid 34 (ObjNum 122 snap 17 seq_num 122) 2024-04-26T01:08:33.072 INFO:tasks.rados.rados.0.smithi164.stdout:276: finishing write tid 1 to smithi16428812-38 2024-04-26T01:08:33.072 INFO:tasks.rados.rados.0.smithi164.stdout:271: done (3 left) 2024-04-26T01:08:33.072 INFO:tasks.rados.rados.0.smithi164.stdout:273: done (2 left) 2024-04-26T01:08:33.072 INFO:tasks.rados.rados.0.smithi164.stdout:278: snap_create 2024-04-26T01:08:33.073 INFO:tasks.rados.rados.0.smithi164.stdout:276: finishing write tid 2 to smithi16428812-38 2024-04-26T01:08:33.082 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-26T01:08:33.084 INFO:tasks.rados.rados.0.smithi164.stdout:276: finishing write tid 3 to smithi16428812-38 2024-04-26T01:08:33.084 INFO:tasks.rados.rados.0.smithi164.stdout:276: finishing write tid 4 to smithi16428812-38 2024-04-26T01:08:33.084 INFO:tasks.rados.rados.0.smithi164.stdout:276: finishing write tid 5 to smithi16428812-38 2024-04-26T01:08:33.089 INFO:tasks.rados.rados.0.smithi164.stdout:276: finishing write tid 6 to smithi16428812-38 2024-04-26T01:08:33.089 INFO:tasks.rados.rados.0.smithi164.stdout:276: oid 38 updating version 0 to 53 2024-04-26T01:08:33.089 INFO:tasks.rados.rados.0.smithi164.stdout:276: oid 38 version 53 is already newer than 52 2024-04-26T01:08:33.089 INFO:tasks.rados.rados.0.smithi164.stdout:276: oid 38 version 53 is already newer than 50 2024-04-26T01:08:33.089 INFO:tasks.rados.rados.0.smithi164.stdout:276: oid 38 updating version 53 to 54 2024-04-26T01:08:33.090 INFO:tasks.rados.rados.0.smithi164.stdout:276: oid 38 version 54 is already newer than 51 2024-04-26T01:08:33.090 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 38 v 54 (ObjNum 124 snap 17 seq_num 124) dirty exists 2024-04-26T01:08:33.090 INFO:tasks.rados.rados.0.smithi164.stdout:276: left oid 38 (ObjNum 124 snap 17 seq_num 124) 2024-04-26T01:08:33.105 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:33.103+0000 7ff2871de640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 32810) UID: 0 2024-04-26T01:08:33.118 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:33.118 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:33.119 INFO:tasks.rados.rados.0.smithi164.stdout:277: finishing copy_from racing read to smithi16428812-7 2024-04-26T01:08:33.132 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:08:33.160 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:33.160 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:33.222 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:33.220+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:33.296 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:33.296 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:33.308 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:33.305+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:33.311 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:08:33.410 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:33.407+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:33.428 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:33.428 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:33.442 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:08:33.511 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:33.509+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:33.562 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:33.562 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:33.576 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:08:33.612 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:33.611+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:33.614 INFO:tasks.rados.rados.0.smithi164.stdout:277: finishing copy_from to smithi16428812-7 2024-04-26T01:08:33.614 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 7 v 47 (ObjNum 118 snap 15 seq_num 118) dirty exists 2024-04-26T01:08:33.705 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:33.705 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:33.715 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:33.715+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:33.719 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:08:33.817 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:33.815+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:33.846 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:33.846 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:33.862 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:08:33.917 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:33.913+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:33.988 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:33.989 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:34.003 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:08:34.019 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:34.015+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:34.045 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.3 to 1 (10000) 2024-04-26T01:08:34.048 INFO:tasks.rados.rados.0.smithi164.stdout:276: done (2 left) 2024-04-26T01:08:34.048 INFO:tasks.rados.rados.0.smithi164.stdout:277: done (1 left) 2024-04-26T01:08:34.048 INFO:tasks.rados.rados.0.smithi164.stdout:278: done (0 left) 2024-04-26T01:08:34.049 INFO:tasks.rados.rados.0.smithi164.stdout:279: snap_create 2024-04-26T01:08:34.067 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-26T01:08:34.120 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:34.119+0000 7ff2871de640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 32810) UID: 0 2024-04-26T01:08:34.131 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:34.132 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:34.147 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:08:34.221 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:34.217+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:34.312 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:34.312 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:34.321 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:34.319+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:34.330 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:08:34.422 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:34.417+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:34.454 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:34.454 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:34.468 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:08:34.522 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:34.520+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:34.598 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:34.598 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:34.612 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:08:34.654 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:34.652+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:34.725 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:34.725+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:34.782 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:34.782 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:34.796 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:08:34.842 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:34.840+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:34.925 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:34.925 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:34.927 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:34.927+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:34.940 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:08:35.028 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:35.025+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:35.048 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.4 to 1 (10000) 2024-04-26T01:08:35.051 INFO:tasks.rados.rados.0.smithi164.stdout:279: done (0 left) 2024-04-26T01:08:35.051 INFO:tasks.rados.rados.0.smithi164.stdout:280: rollback oid 6 current snap is 19 2024-04-26T01:08:35.051 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 6 to 17 2024-04-26T01:08:35.051 INFO:tasks.rados.rados.0.smithi164.stdout:281: read oid 17 snap -1 2024-04-26T01:08:35.052 INFO:tasks.rados.rados.0.smithi164.stdout:281: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-26T01:08:35.052 INFO:tasks.rados.rados.0.smithi164.stdout:282: delete oid 30 current snap is 19 2024-04-26T01:08:35.062 INFO:tasks.rados.rados.0.smithi164.stdout:280: finishing rollback tid 0 to smithi16428812-6 2024-04-26T01:08:35.064 INFO:tasks.rados.rados.0.smithi164.stdout:280: finishing rollback tid 1 to smithi16428812-6 2024-04-26T01:08:35.064 INFO:tasks.rados.rados.0.smithi164.stdout:280: finishing rollback tid 2 to smithi16428812-6 2024-04-26T01:08:35.064 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 6 v 57 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-26T01:08:35.065 INFO:tasks.rados.rados.0.smithi164.stdout:280: done (2 left) 2024-04-26T01:08:35.065 INFO:tasks.rados.rados.0.smithi164.stdout:282: done (1 left) 2024-04-26T01:08:35.065 INFO:tasks.rados.rados.0.smithi164.stdout:283: snap_remove snap 17 2024-04-26T01:08:35.077 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-26T01:08:35.082 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-26T01:08:35.121 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:35.121 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:35.129 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:35.125+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:35.137 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:08:35.179 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:08:35.230 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:35.231+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:35.273 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:35.274 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:35.288 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:08:35.331 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:35.328+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:35.335 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:35.335 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:35.357 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:35.358 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:35.383 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:35.383 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:35.408 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:35.408 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:35.414 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:35.414 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:35.414 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:08:35.414 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:08:35.414 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:35.429 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:08:35.432 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:35.431+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:35.432 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:35.432 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:35.457 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:35.457 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:35.485 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:35.485 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:35.516 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:35.516 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:35.532 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:35.529+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:35.544 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:35.545 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:35.557 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:35.557 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:08:35.557 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:08:35.557 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:08:35.557 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.557 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.2s5 32/30 ECSubWrite(tid=157, reqid=client.4521.0:13, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:35.558 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.308314+0000", 2024-04-26T01:08:35.558 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 36.248804282999998, 2024-04-26T01:08:35.558 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011288362999999999, 2024-04-26T01:08:35.558 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.558 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.558 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.558 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.558 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.558 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.558 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308314+0000", 2024-04-26T01:08:35.558 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.558 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.559 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.559 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.559 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308314+0000", 2024-04-26T01:08:35.559 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.559 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.559 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.559 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.559 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308315+0000", 2024-04-26T01:08:35.559 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.6599999999999995e-07 2024-04-26T01:08:35.559 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.559 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.559 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.559 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308467+0000", 2024-04-26T01:08:35.560 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000152249 2024-04-26T01:08:35.560 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.560 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.560 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.560 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308602+0000", 2024-04-26T01:08:35.560 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000135048 2024-04-26T01:08:35.560 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.560 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.560 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.560 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308653+0000", 2024-04-26T01:08:35.560 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0864e-05 2024-04-26T01:08:35.560 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.560 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.560 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.561 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308834+0000", 2024-04-26T01:08:35.561 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000180466 2024-04-26T01:08:35.561 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.561 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.561 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.561 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308857+0000", 2024-04-26T01:08:35.561 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2904e-05 2024-04-26T01:08:35.561 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.562 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.562 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.562 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.319506+0000", 2024-04-26T01:08:35.562 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010649141000000001 2024-04-26T01:08:35.562 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.562 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.562 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.562 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.319603+0000", 2024-04-26T01:08:35.562 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6924999999999998e-05 2024-04-26T01:08:35.562 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.563 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.563 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.563 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.563 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.563 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 32/30 ECSubWrite(tid=0, reqid=client.4521.0:16, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:35.563 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:35.563 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 36.246585963999998, 2024-04-26T01:08:35.563 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012119314000000001, 2024-04-26T01:08:35.563 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.563 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.563 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.563 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.563 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.564 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.564 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:35.564 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.564 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.564 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.564 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.564 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:35.564 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.564 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.564 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.564 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.564 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:35.564 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.7000000000000004e-07 2024-04-26T01:08:35.565 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.565 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.565 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.565 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310733+0000", 2024-04-26T01:08:35.565 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019985599999999999 2024-04-26T01:08:35.565 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.565 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.565 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.565 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310792+0000", 2024-04-26T01:08:35.565 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.8894000000000002e-05 2024-04-26T01:08:35.565 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.565 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.566 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.566 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310816+0000", 2024-04-26T01:08:35.566 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4094e-05 2024-04-26T01:08:35.566 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.566 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.566 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.566 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311022+0000", 2024-04-26T01:08:35.566 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00020608599999999999 2024-04-26T01:08:35.566 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.566 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.566 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.566 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311044+0000", 2024-04-26T01:08:35.566 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1889999999999999e-05 2024-04-26T01:08:35.566 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.567 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.567 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.567 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322407+0000", 2024-04-26T01:08:35.567 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011362778 2024-04-26T01:08:35.567 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.567 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.567 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.567 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322652+0000", 2024-04-26T01:08:35.567 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00024504600000000001 2024-04-26T01:08:35.567 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.567 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.567 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.567 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.567 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.568 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.es2 33/30 ECSubWrite(tid=0, reqid=client.4521.0:22, at_version=33'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:35.568 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.311344+0000", 2024-04-26T01:08:35.568 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 36.245774449999999, 2024-04-26T01:08:35.568 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010979113, 2024-04-26T01:08:35.568 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.568 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.568 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.568 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.568 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.568 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.568 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311344+0000", 2024-04-26T01:08:35.568 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.568 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.569 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.569 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.569 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311344+0000", 2024-04-26T01:08:35.569 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.569 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.569 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.569 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.569 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311351+0000", 2024-04-26T01:08:35.569 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1040000000000001e-06 2024-04-26T01:08:35.570 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.570 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.570 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.570 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311496+0000", 2024-04-26T01:08:35.570 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000144852 2024-04-26T01:08:35.570 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.570 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.570 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.570 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311639+0000", 2024-04-26T01:08:35.570 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014267800000000001 2024-04-26T01:08:35.570 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.570 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.570 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.571 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311663+0000", 2024-04-26T01:08:35.571 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4090000000000001e-05 2024-04-26T01:08:35.571 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.571 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.571 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.571 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311836+0000", 2024-04-26T01:08:35.571 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000173349 2024-04-26T01:08:35.571 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.571 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.571 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.571 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311859+0000", 2024-04-26T01:08:35.571 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2504e-05 2024-04-26T01:08:35.571 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.571 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.572 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.572 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322193+0000", 2024-04-26T01:08:35.572 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010334157 2024-04-26T01:08:35.572 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.572 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.572 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.572 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322323+0000", 2024-04-26T01:08:35.572 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013037899999999999 2024-04-26T01:08:35.572 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.572 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.572 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.572 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.572 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.572 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.8s3 32/30 ECSubWrite(tid=1, reqid=client.4521.0:8, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:35.573 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.312999+0000", 2024-04-26T01:08:35.573 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 36.244119988000001, 2024-04-26T01:08:35.573 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0094850960000000002, 2024-04-26T01:08:35.573 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.573 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.573 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.573 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.573 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.573 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.573 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312999+0000", 2024-04-26T01:08:35.573 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.573 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.573 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.573 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.574 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312999+0000", 2024-04-26T01:08:35.574 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.574 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.574 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.574 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.574 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313001+0000", 2024-04-26T01:08:35.574 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.21e-06 2024-04-26T01:08:35.574 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.574 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.574 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.574 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313076+0000", 2024-04-26T01:08:35.574 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.5525000000000006e-05 2024-04-26T01:08:35.574 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.574 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.575 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.575 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313194+0000", 2024-04-26T01:08:35.575 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000118004 2024-04-26T01:08:35.575 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.575 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.575 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.575 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313238+0000", 2024-04-26T01:08:35.575 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3523999999999999e-05 2024-04-26T01:08:35.575 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.575 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.575 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.576 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313570+0000", 2024-04-26T01:08:35.576 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033178100000000001 2024-04-26T01:08:35.576 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.576 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.576 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.576 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313595+0000", 2024-04-26T01:08:35.576 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5181000000000002e-05 2024-04-26T01:08:35.576 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.576 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.576 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.576 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322175+0000", 2024-04-26T01:08:35.576 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0085803070000000006 2024-04-26T01:08:35.576 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.576 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.577 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.577 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322484+0000", 2024-04-26T01:08:35.577 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000308564 2024-04-26T01:08:35.577 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.577 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.577 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.577 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.577 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.577 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.es2 33/30 ECSubWrite(tid=1, reqid=client.4521.0:23, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:35.577 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.313493+0000", 2024-04-26T01:08:35.577 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 36.243625954999999, 2024-04-26T01:08:35.577 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0089452720000000006, 2024-04-26T01:08:35.577 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.577 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.578 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.578 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.578 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.578 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.578 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313493+0000", 2024-04-26T01:08:35.578 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.578 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.578 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.578 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.578 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313493+0000", 2024-04-26T01:08:35.578 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.578 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.578 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.579 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.579 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313494+0000", 2024-04-26T01:08:35.579 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5039999999999999e-06 2024-04-26T01:08:35.579 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.579 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.579 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.579 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313593+0000", 2024-04-26T01:08:35.579 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.9355e-05 2024-04-26T01:08:35.579 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.579 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.579 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.579 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313756+0000", 2024-04-26T01:08:35.580 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000162762 2024-04-26T01:08:35.580 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.580 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.580 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.580 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313781+0000", 2024-04-26T01:08:35.580 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4893e-05 2024-04-26T01:08:35.580 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.580 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.580 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.580 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313944+0000", 2024-04-26T01:08:35.580 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016297600000000001 2024-04-26T01:08:35.580 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.580 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.580 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.581 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313968+0000", 2024-04-26T01:08:35.581 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3986999999999999e-05 2024-04-26T01:08:35.581 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.581 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.581 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.581 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322340+0000", 2024-04-26T01:08:35.581 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0083720840000000001 2024-04-26T01:08:35.581 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.581 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.581 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.581 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322438+0000", 2024-04-26T01:08:35.581 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7711e-05 2024-04-26T01:08:35.581 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.581 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.582 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.582 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.582 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.582 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 50/30 ECSubWrite(tid=29, reqid=osd.8.0:29, at_version=50'18, trim_to=0'0, roll_forward_to=47'16))", 2024-04-26T01:08:35.582 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.040723+0000", 2024-04-26T01:08:35.582 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 19.516395715000002, 2024-04-26T01:08:35.582 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012574701000000001, 2024-04-26T01:08:35.582 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.582 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.582 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.582 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.582 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040723+0000", 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040723+0000", 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040724+0000", 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.37e-07 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.583 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.584 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040778+0000", 2024-04-26T01:08:35.584 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.4453000000000002e-05 2024-04-26T01:08:35.584 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.584 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.584 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.584 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040860+0000", 2024-04-26T01:08:35.584 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.2060999999999999e-05 2024-04-26T01:08:35.584 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.584 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.584 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.584 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040883+0000", 2024-04-26T01:08:35.584 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2994e-05 2024-04-26T01:08:35.584 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.584 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.585 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.585 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.041058+0000", 2024-04-26T01:08:35.585 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017496999999999999 2024-04-26T01:08:35.585 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.585 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.585 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.585 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.041082+0000", 2024-04-26T01:08:35.585 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.393e-05 2024-04-26T01:08:35.585 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.585 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.585 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.585 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053181+0000", 2024-04-26T01:08:35.585 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012099085000000001 2024-04-26T01:08:35.586 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.586 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.586 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.586 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053298+0000", 2024-04-26T01:08:35.586 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000116471 2024-04-26T01:08:35.586 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.586 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.586 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.586 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.586 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.586 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 50/30 ECSubWrite(tid=214, reqid=osd.4.0:214, at_version=50'22, trim_to=0'0, roll_forward_to=45'20))", 2024-04-26T01:08:35.586 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.047877+0000", 2024-04-26T01:08:35.586 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 19.509241051, 2024-04-26T01:08:35.587 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010720746, 2024-04-26T01:08:35.587 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.587 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.587 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.587 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.587 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.587 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.587 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047877+0000", 2024-04-26T01:08:35.587 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.587 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.587 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.587 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.587 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047877+0000", 2024-04-26T01:08:35.587 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.588 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.588 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.588 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.588 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047878+0000", 2024-04-26T01:08:35.588 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.6699999999999999e-07 2024-04-26T01:08:35.588 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.588 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.588 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.588 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047930+0000", 2024-04-26T01:08:35.588 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.1669999999999998e-05 2024-04-26T01:08:35.588 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.588 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.588 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.588 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048154+0000", 2024-04-26T01:08:35.589 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022397399999999999 2024-04-26T01:08:35.589 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.589 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.589 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.589 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048176+0000", 2024-04-26T01:08:35.589 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2734e-05 2024-04-26T01:08:35.589 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.589 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.589 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.589 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048452+0000", 2024-04-26T01:08:35.589 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00027581799999999999 2024-04-26T01:08:35.589 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.589 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.589 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.589 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048478+0000", 2024-04-26T01:08:35.590 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5723000000000001e-05 2024-04-26T01:08:35.590 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.590 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.590 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.590 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058506+0000", 2024-04-26T01:08:35.590 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010027635999999999 2024-04-26T01:08:35.590 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.590 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.590 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.590 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058598+0000", 2024-04-26T01:08:35.590 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2724000000000002e-05 2024-04-26T01:08:35.590 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.590 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.591 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.591 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.591 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.591 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.6s2 50/30 ECSubWrite(tid=41, reqid=osd.0.0:41, at_version=50'24, trim_to=0'0, roll_forward_to=45'22))", 2024-04-26T01:08:35.591 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.052765+0000", 2024-04-26T01:08:35.591 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 19.504353306999999, 2024-04-26T01:08:35.591 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0080835920000000006, 2024-04-26T01:08:35.591 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.591 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.591 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.591 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.591 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.591 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.591 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052765+0000", 2024-04-26T01:08:35.591 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052765+0000", 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052766+0000", 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1999999999999999e-07 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052820+0000", 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.4441000000000003e-05 2024-04-26T01:08:35.592 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.593 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.593 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.593 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052916+0000", 2024-04-26T01:08:35.593 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5597e-05 2024-04-26T01:08:35.593 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.593 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.593 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.593 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052940+0000", 2024-04-26T01:08:35.593 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3626e-05 2024-04-26T01:08:35.593 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.593 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.593 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.593 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053100+0000", 2024-04-26T01:08:35.593 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015994800000000001 2024-04-26T01:08:35.593 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.594 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.594 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.594 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053126+0000", 2024-04-26T01:08:35.594 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6393e-05 2024-04-26T01:08:35.594 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.594 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.594 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.594 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060748+0000", 2024-04-26T01:08:35.594 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.007622523 2024-04-26T01:08:35.594 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.594 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.594 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.594 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060849+0000", 2024-04-26T01:08:35.594 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000100344 2024-04-26T01:08:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 55/30 ECSubWrite(tid=254, reqid=client.4521.0:549, at_version=55'29, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:21.088071+0000", 2024-04-26T01:08:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 14.469047113, 2024-04-26T01:08:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.023607655000000002, 2024-04-26T01:08:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.596 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.088071+0000", 2024-04-26T01:08:35.596 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.596 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.596 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.596 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.596 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.088071+0000", 2024-04-26T01:08:35.596 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.596 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.596 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.596 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.596 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.088074+0000", 2024-04-26T01:08:35.596 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.097e-06 2024-04-26T01:08:35.596 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.596 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.597 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.597 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.088360+0000", 2024-04-26T01:08:35.597 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000286261 2024-04-26T01:08:35.597 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.597 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.597 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.597 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.103040+0000", 2024-04-26T01:08:35.597 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.014680442 2024-04-26T01:08:35.597 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.597 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.597 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.597 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.103049+0000", 2024-04-26T01:08:35.597 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.251e-06 2024-04-26T01:08:35.597 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.598 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.598 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.598 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.103185+0000", 2024-04-26T01:08:35.598 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013595400000000001 2024-04-26T01:08:35.598 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.598 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.598 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.598 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.103212+0000", 2024-04-26T01:08:35.598 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6517e-05 2024-04-26T01:08:35.598 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.598 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.598 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.598 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.111554+0000", 2024-04-26T01:08:35.599 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.008342426 2024-04-26T01:08:35.599 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.599 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.599 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.599 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.111679+0000", 2024-04-26T01:08:35.599 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000124707 2024-04-26T01:08:35.599 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.599 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.599 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.599 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.599 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.599 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 55/30 ECSubWrite(tid=256, reqid=client.4521.0:550, at_version=55'30, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:35.599 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:21.102883+0000", 2024-04-26T01:08:35.599 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 14.454235260000001, 2024-04-26T01:08:35.599 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0094720329999999995, 2024-04-26T01:08:35.600 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.600 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.600 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.600 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.600 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.600 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.600 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.102883+0000", 2024-04-26T01:08:35.600 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.600 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.600 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.600 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.600 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.102883+0000", 2024-04-26T01:08:35.601 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.601 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.601 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.601 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.601 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.102884+0000", 2024-04-26T01:08:35.601 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8000000000000005e-07 2024-04-26T01:08:35.601 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.601 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.601 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.601 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.103016+0000", 2024-04-26T01:08:35.601 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013183699999999999 2024-04-26T01:08:35.601 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.601 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.602 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.602 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105250+0000", 2024-04-26T01:08:35.602 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002234307 2024-04-26T01:08:35.602 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.602 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.602 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.602 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105272+0000", 2024-04-26T01:08:35.602 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1886e-05 2024-04-26T01:08:35.602 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.602 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.602 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.602 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105352+0000", 2024-04-26T01:08:35.603 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.0383999999999995e-05 2024-04-26T01:08:35.603 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.603 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.603 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.603 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105364+0000", 2024-04-26T01:08:35.603 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1331999999999999e-05 2024-04-26T01:08:35.603 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.603 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.603 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.603 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.112272+0000", 2024-04-26T01:08:35.603 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0069084970000000004 2024-04-26T01:08:35.603 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.603 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.603 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.604 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.112355+0000", 2024-04-26T01:08:35.604 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.3010000000000007e-05 2024-04-26T01:08:35.604 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.604 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.604 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.604 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.604 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.604 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 55/30 ECSubWrite(tid=257, reqid=client.4521.0:551, at_version=55'31, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:35.604 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:21.105099+0000", 2024-04-26T01:08:35.604 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 14.452019163999999, 2024-04-26T01:08:35.604 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0073293309999999997, 2024-04-26T01:08:35.604 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.604 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.604 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.605 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.605 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.605 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.605 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105099+0000", 2024-04-26T01:08:35.605 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.605 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.605 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.605 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.605 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105099+0000", 2024-04-26T01:08:35.605 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.605 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.605 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.605 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.605 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105101+0000", 2024-04-26T01:08:35.605 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.75e-06 2024-04-26T01:08:35.606 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.606 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.606 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.606 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105182+0000", 2024-04-26T01:08:35.606 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.0533999999999998e-05 2024-04-26T01:08:35.606 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.606 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.606 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.606 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107687+0000", 2024-04-26T01:08:35.606 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0025055659999999999 2024-04-26T01:08:35.606 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.606 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.607 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.607 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107716+0000", 2024-04-26T01:08:35.607 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8750000000000001e-05 2024-04-26T01:08:35.607 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.607 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.607 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.607 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107877+0000", 2024-04-26T01:08:35.607 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016068300000000001 2024-04-26T01:08:35.607 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.607 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.607 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.607 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107902+0000", 2024-04-26T01:08:35.608 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.561e-05 2024-04-26T01:08:35.608 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.608 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.608 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.608 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.112364+0000", 2024-04-26T01:08:35.608 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0044621039999999997 2024-04-26T01:08:35.608 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.608 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.608 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.608 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.112429+0000", 2024-04-26T01:08:35.608 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.4333999999999998e-05 2024-04-26T01:08:35.608 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.608 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.609 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.609 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.609 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.609 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 55/30 ECSubWrite(tid=259, reqid=client.4521.0:552, at_version=55'32, trim_to=0'0, roll_forward_to=55'28))", 2024-04-26T01:08:35.609 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:21.107592+0000", 2024-04-26T01:08:35.609 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 14.449526798000001, 2024-04-26T01:08:35.609 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0078481520000000006, 2024-04-26T01:08:35.609 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.609 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.609 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.609 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.609 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.609 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.610 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107592+0000", 2024-04-26T01:08:35.610 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.610 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.610 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.610 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.610 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107592+0000", 2024-04-26T01:08:35.610 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.610 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.610 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.610 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.610 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107594+0000", 2024-04-26T01:08:35.610 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.764e-06 2024-04-26T01:08:35.611 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.611 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.611 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.611 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107640+0000", 2024-04-26T01:08:35.611 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.6866000000000002e-05 2024-04-26T01:08:35.611 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.611 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.611 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.611 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.113932+0000", 2024-04-26T01:08:35.611 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0062914210000000002 2024-04-26T01:08:35.611 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.611 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.611 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.611 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.113942+0000", 2024-04-26T01:08:35.612 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0627e-05 2024-04-26T01:08:35.612 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.612 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.612 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.612 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.114006+0000", 2024-04-26T01:08:35.612 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.3448999999999995e-05 2024-04-26T01:08:35.612 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.612 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.612 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.612 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.114016+0000", 2024-04-26T01:08:35.612 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.9550000000000004e-06 2024-04-26T01:08:35.612 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.612 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.613 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.613 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.115404+0000", 2024-04-26T01:08:35.613 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0013883980000000001 2024-04-26T01:08:35.613 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.613 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.613 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.613 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.115440+0000", 2024-04-26T01:08:35.613 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.5672e-05 2024-04-26T01:08:35.613 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.613 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.613 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.613 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.613 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.613 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpRead(3.bs2 55/30 ECSubRead(tid=262, to_read={3:d5691d7a:::smithi16428812-40:head=0,438272,0}, subchunks={3:d5691d7a:::smithi16428812-40:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:35.614 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:21.113903+0000", 2024-04-26T01:08:35.614 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 14.443215209, 2024-04-26T01:08:35.614 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0071693190000000004, 2024-04-26T01:08:35.614 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.614 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.614 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.614 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.614 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.614 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.614 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.113903+0000", 2024-04-26T01:08:35.614 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.614 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.614 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.615 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.615 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.113903+0000", 2024-04-26T01:08:35.615 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.615 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.615 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.615 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.615 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.113904+0000", 2024-04-26T01:08:35.615 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8900000000000001e-07 2024-04-26T01:08:35.615 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.615 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.615 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.615 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.113912+0000", 2024-04-26T01:08:35.615 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.9540000000000004e-06 2024-04-26T01:08:35.615 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.616 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.616 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.616 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.120147+0000", 2024-04-26T01:08:35.616 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0062352040000000003 2024-04-26T01:08:35.616 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.616 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.616 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.616 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.120175+0000", 2024-04-26T01:08:35.616 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.83e-05 2024-04-26T01:08:35.616 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.616 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.616 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.617 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.120260+0000", 2024-04-26T01:08:35.617 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.4646000000000002e-05 2024-04-26T01:08:35.617 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.617 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.617 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.617 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.121073+0000", 2024-04-26T01:08:35.617 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00081292599999999995 2024-04-26T01:08:35.617 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.617 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.617 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.617 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.617 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.617 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s3 58/30 ECSubWrite(tid=62, reqid=osd.14.0:62, at_version=58'47, trim_to=0'0, roll_forward_to=56'45))", 2024-04-26T01:08:35.618 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:24.166131+0000", 2024-04-26T01:08:35.618 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 11.390987537999999, 2024-04-26T01:08:35.618 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0076169009999999997, 2024-04-26T01:08:35.618 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.618 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.618 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.618 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.618 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.618 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.618 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166131+0000", 2024-04-26T01:08:35.618 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.618 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.618 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.618 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.619 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166131+0000", 2024-04-26T01:08:35.619 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.619 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.619 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.619 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.619 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166132+0000", 2024-04-26T01:08:35.619 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.9299999999999998e-07 2024-04-26T01:08:35.619 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.619 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.619 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.619 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166185+0000", 2024-04-26T01:08:35.619 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.3712999999999999e-05 2024-04-26T01:08:35.619 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.620 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.620 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.620 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166289+0000", 2024-04-26T01:08:35.620 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000103391 2024-04-26T01:08:35.620 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.620 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.620 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.620 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166311+0000", 2024-04-26T01:08:35.620 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2359999999999999e-05 2024-04-26T01:08:35.620 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.620 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.620 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.620 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166470+0000", 2024-04-26T01:08:35.620 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000158543 2024-04-26T01:08:35.621 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.621 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.621 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.621 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166499+0000", 2024-04-26T01:08:35.621 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9074e-05 2024-04-26T01:08:35.621 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.621 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.621 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.621 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.173647+0000", 2024-04-26T01:08:35.621 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0071487269999999997 2024-04-26T01:08:35.621 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.621 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.621 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.622 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.173748+0000", 2024-04-26T01:08:35.622 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0001005 2024-04-26T01:08:35.622 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.622 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.622 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.622 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.622 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.622 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 59/30 ECSubWrite(tid=320, reqid=osd.4.0:320, at_version=59'43, trim_to=0'0, roll_forward_to=58'41))", 2024-04-26T01:08:35.622 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:25.158834+0000", 2024-04-26T01:08:35.622 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 10.398284877, 2024-04-26T01:08:35.622 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0071564360000000004, 2024-04-26T01:08:35.622 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.622 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.623 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.623 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.623 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.623 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.623 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.158834+0000", 2024-04-26T01:08:35.623 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.623 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.623 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.623 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.623 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.158834+0000", 2024-04-26T01:08:35.623 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.623 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.623 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.623 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.624 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.158834+0000", 2024-04-26T01:08:35.624 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.2999999999999999e-07 2024-04-26T01:08:35.624 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.624 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.624 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.624 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.158893+0000", 2024-04-26T01:08:35.624 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.808e-05 2024-04-26T01:08:35.624 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.624 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.624 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.624 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.159012+0000", 2024-04-26T01:08:35.624 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000119657 2024-04-26T01:08:35.624 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.624 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.625 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.625 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.159039+0000", 2024-04-26T01:08:35.625 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7189999999999999e-05 2024-04-26T01:08:35.625 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.625 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.625 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.625 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.159216+0000", 2024-04-26T01:08:35.625 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017637600000000001 2024-04-26T01:08:35.625 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.625 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.625 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.625 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.159242+0000", 2024-04-26T01:08:35.625 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6239999999999999e-05 2024-04-26T01:08:35.625 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.625 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.626 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.626 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.165863+0000", 2024-04-26T01:08:35.626 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0066210829999999998 2024-04-26T01:08:35.626 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.626 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.626 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.626 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.165990+0000", 2024-04-26T01:08:35.626 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012698000000000001 2024-04-26T01:08:35.626 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.626 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.626 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.627 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.627 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.627 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s3 59/30 ECSubWrite(tid=66, reqid=osd.14.0:66, at_version=59'51, trim_to=0'0, roll_forward_to=58'49))", 2024-04-26T01:08:35.627 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:25.171370+0000", 2024-04-26T01:08:35.627 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 10.385748591, 2024-04-26T01:08:35.627 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00777277, 2024-04-26T01:08:35.627 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.627 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.627 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.627 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.627 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.627 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.627 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.171370+0000", 2024-04-26T01:08:35.627 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.628 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.628 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.628 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.628 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.171370+0000", 2024-04-26T01:08:35.628 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.628 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.628 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.628 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.628 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.171371+0000", 2024-04-26T01:08:35.628 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.31e-06 2024-04-26T01:08:35.628 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.628 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.628 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.628 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.171423+0000", 2024-04-26T01:08:35.629 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.1323000000000003e-05 2024-04-26T01:08:35.629 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.629 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.629 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.629 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.171527+0000", 2024-04-26T01:08:35.629 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000103984 2024-04-26T01:08:35.629 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.629 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.629 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.629 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.171551+0000", 2024-04-26T01:08:35.629 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4470000000000001e-05 2024-04-26T01:08:35.629 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.629 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.629 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.630 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.173214+0000", 2024-04-26T01:08:35.630 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0016628050000000001 2024-04-26T01:08:35.630 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.630 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.630 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.630 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.173261+0000", 2024-04-26T01:08:35.630 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.7273e-05 2024-04-26T01:08:35.630 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.630 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.630 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.630 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.179091+0000", 2024-04-26T01:08:35.630 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0058294380000000002 2024-04-26T01:08:35.630 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.631 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.631 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.631 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.179143+0000", 2024-04-26T01:08:35.631 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.2166999999999997e-05 2024-04-26T01:08:35.631 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.631 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.631 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.631 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.631 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.631 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 59/30 ECSubWrite(tid=53, reqid=client.4521.0:613, at_version=59'38, trim_to=0'0, roll_forward_to=59'36))", 2024-04-26T01:08:35.631 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:25.401515+0000", 2024-04-26T01:08:35.631 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 10.155603072, 2024-04-26T01:08:35.631 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0070293689999999997, 2024-04-26T01:08:35.632 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.632 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.632 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.632 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.632 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.632 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.632 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401515+0000", 2024-04-26T01:08:35.632 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.632 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.632 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.632 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.632 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401515+0000", 2024-04-26T01:08:35.632 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.632 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.632 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.633 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.633 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401517+0000", 2024-04-26T01:08:35.633 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.917e-06 2024-04-26T01:08:35.633 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.633 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.633 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.633 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401559+0000", 2024-04-26T01:08:35.633 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.1180000000000002e-05 2024-04-26T01:08:35.633 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.633 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.633 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.633 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401650+0000", 2024-04-26T01:08:35.633 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.1663000000000007e-05 2024-04-26T01:08:35.633 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.634 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.634 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.634 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401666+0000", 2024-04-26T01:08:35.634 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5583999999999999e-05 2024-04-26T01:08:35.634 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.634 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.634 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.634 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401848+0000", 2024-04-26T01:08:35.634 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00018238999999999999 2024-04-26T01:08:35.634 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.634 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.634 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.634 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401880+0000", 2024-04-26T01:08:35.635 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.1446999999999999e-05 2024-04-26T01:08:35.635 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.635 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.635 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.635 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.408444+0000", 2024-04-26T01:08:35.635 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0065639050000000001 2024-04-26T01:08:35.635 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.635 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.635 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.635 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.408545+0000", 2024-04-26T01:08:35.635 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000101283 2024-04-26T01:08:35.635 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.635 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.635 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.636 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 60/30 ECSubWrite(tid=349, reqid=osd.4.0:349, at_version=60'52, trim_to=0'0, roll_forward_to=59'50))", 2024-04-26T01:08:35.636 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:26.179457+0000", 2024-04-26T01:08:35.636 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.3776612860000004, 2024-04-26T01:08:35.636 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0076187240000000003, 2024-04-26T01:08:35.636 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.636 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.636 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.636 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.636 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.636 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179457+0000", 2024-04-26T01:08:35.636 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.637 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.637 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.637 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179457+0000", 2024-04-26T01:08:35.637 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.637 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.637 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.637 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179458+0000", 2024-04-26T01:08:35.637 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3700000000000001e-07 2024-04-26T01:08:35.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.637 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.637 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.637 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179498+0000", 2024-04-26T01:08:35.637 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.0726999999999998e-05 2024-04-26T01:08:35.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.638 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.638 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.638 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179634+0000", 2024-04-26T01:08:35.638 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013533299999999999 2024-04-26T01:08:35.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.638 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.638 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.638 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179655+0000", 2024-04-26T01:08:35.638 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1710000000000001e-05 2024-04-26T01:08:35.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.638 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.638 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.638 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179855+0000", 2024-04-26T01:08:35.639 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019919 2024-04-26T01:08:35.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.639 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.639 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.639 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179882+0000", 2024-04-26T01:08:35.639 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7250000000000002e-05 2024-04-26T01:08:35.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.639 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.639 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.639 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.186989+0000", 2024-04-26T01:08:35.639 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0071068729999999997 2024-04-26T01:08:35.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.639 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.639 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.639 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.187076+0000", 2024-04-26T01:08:35.640 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.7204000000000001e-05 2024-04-26T01:08:35.640 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.640 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.640 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.640 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.640 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 60/30 ECSubWrite(tid=351, reqid=osd.4.0:351, at_version=60'54, trim_to=0'0, roll_forward_to=59'50))", 2024-04-26T01:08:35.640 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:26.179502+0000", 2024-04-26T01:08:35.640 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.3776165159999998, 2024-04-26T01:08:35.640 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.009331499, 2024-04-26T01:08:35.640 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.640 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.640 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.640 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.641 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.641 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.641 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179502+0000", 2024-04-26T01:08:35.641 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.641 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.641 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.641 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.641 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179502+0000", 2024-04-26T01:08:35.641 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.641 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.641 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.641 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.641 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179502+0000", 2024-04-26T01:08:35.641 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.7e-07 2024-04-26T01:08:35.641 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.642 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.642 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.642 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179532+0000", 2024-04-26T01:08:35.642 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9977000000000001e-05 2024-04-26T01:08:35.642 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.642 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.642 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.642 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179726+0000", 2024-04-26T01:08:35.642 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019319300000000001 2024-04-26T01:08:35.642 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.642 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.642 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.642 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179745+0000", 2024-04-26T01:08:35.642 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9167000000000001e-05 2024-04-26T01:08:35.643 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.643 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.643 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.643 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.183638+0000", 2024-04-26T01:08:35.643 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0038929580000000002 2024-04-26T01:08:35.643 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.643 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.643 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.643 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.183663+0000", 2024-04-26T01:08:35.643 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5673999999999998e-05 2024-04-26T01:08:35.643 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.643 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.643 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.643 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.188734+0000", 2024-04-26T01:08:35.644 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0050702530000000003 2024-04-26T01:08:35.644 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.644 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.644 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.644 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.188834+0000", 2024-04-26T01:08:35.644 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.9807000000000002e-05 2024-04-26T01:08:35.644 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.644 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.644 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.644 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.644 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.644 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 60/30 ECSubWrite(tid=58, reqid=osd.8.0:58, at_version=60'44, trim_to=0'0, roll_forward_to=59'40))", 2024-04-26T01:08:35.644 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:26.179976+0000", 2024-04-26T01:08:35.644 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.3771422619999996, 2024-04-26T01:08:35.644 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0070517189999999997, 2024-04-26T01:08:35.645 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:35.645 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:35.645 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:35.645 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:35.645 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.645 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:35.645 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179976+0000", 2024-04-26T01:08:35.645 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.645 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.645 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.645 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:35.645 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179976+0000", 2024-04-26T01:08:35.646 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:35.646 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.646 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.646 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:35.646 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179978+0000", 2024-04-26T01:08:35.646 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6670000000000001e-06 2024-04-26T01:08:35.646 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.646 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.646 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:35.646 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.180025+0000", 2024-04-26T01:08:35.646 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.7227e-05 2024-04-26T01:08:35.646 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.646 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.180097+0000", 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1366999999999996e-05 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.180114+0000", 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7506000000000001e-05 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.180245+0000", 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013061400000000001 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.647 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:35.648 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.180259+0000", 2024-04-26T01:08:35.648 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.472e-05 2024-04-26T01:08:35.648 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.648 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.648 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:35.648 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.186934+0000", 2024-04-26T01:08:35.648 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.006674681 2024-04-26T01:08:35.648 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:35.648 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:35.648 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:35.648 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.187028+0000", 2024-04-26T01:08:35.648 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.3937000000000005e-05 2024-04-26T01:08:35.648 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.648 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.649 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.649 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:35.649 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:35.649 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:35.650 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:35.650 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:35.651 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:35.651 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:35.652 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:08:35.652 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:35.645+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:35.652 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:08:35.653 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:08:35.662 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:35.662 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:35.692 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:35.692 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:35.725 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:35.720+0000 7fc265d9b640 0 -- 172.21.15.8:0/2816222409 >> [v2:172.21.15.116:6800/2158650188,v1:172.21.15.116:6801/2158650188] conn(0x7fc268222a10 msgr2=0x7fc268224e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:08:35.751 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:35.751+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:35.826 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:35.827 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:35.827 INFO:teuthology.orchestra.run.smithi012.stdout: "num_ops": 0 2024-04-26T01:08:35.827 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:35.841 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:08:35.887 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:35.885+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:35.952 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:35.949+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:35.967 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:35.967 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:35.967 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:08:35.967 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:08:35.967 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:35.982 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:08:36.051 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.5 to 1 (10000) 2024-04-26T01:08:36.052 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-04-26T01:08:36.053 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:36.048+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:36.057 INFO:tasks.rados.rados.0.smithi164.stdout:283: done (1 left) 2024-04-26T01:08:36.057 INFO:tasks.rados.rados.0.smithi164.stdout:284: rollback oid 31 current snap is 19 2024-04-26T01:08:36.057 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 31 to 18 2024-04-26T01:08:36.057 INFO:tasks.rados.rados.0.smithi164.stdout:281: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-26T01:08:36.076 INFO:tasks.rados.rados.0.smithi164.stdout:284: finishing rollback tid 0 to smithi16428812-31 2024-04-26T01:08:36.076 INFO:tasks.rados.rados.0.smithi164.stdout:284: finishing rollback tid 1 to smithi16428812-31 2024-04-26T01:08:36.077 INFO:tasks.rados.rados.0.smithi164.stdout:284: finishing rollback tid 2 to smithi16428812-31 2024-04-26T01:08:36.077 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 31 v 88 (ObjNum 83 snap 5 seq_num 83) dirty exists 2024-04-26T01:08:36.077 INFO:tasks.rados.rados.0.smithi164.stdout:281: done (1 left) 2024-04-26T01:08:36.077 INFO:tasks.rados.rados.0.smithi164.stdout:284: done (0 left) 2024-04-26T01:08:36.077 INFO:tasks.rados.rados.0.smithi164.stdout:285: snap_remove snap 18 2024-04-26T01:08:36.085 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-26T01:08:36.154 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:36.149+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:36.157 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:08:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:08:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:08:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:7 3.8s0 3:1b6b0095:::smithi16428812-3:head [append 0~49152,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.302696+0000", 2024-04-26T01:08:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 36.853427840999998, 2024-04-26T01:08:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.016152680999999999, 2024-04-26T01:08:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 7 2024-04-26T01:08:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.302696+0000", 2024-04-26T01:08:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.302696+0000", 2024-04-26T01:08:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.302704+0000", 2024-04-26T01:08:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1130000000000001e-06 2024-04-26T01:08:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303173+0000", 2024-04-26T01:08:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000469109 2024-04-26T01:08:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303207+0000", 2024-04-26T01:08:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.3937999999999999e-05 2024-04-26T01:08:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303229+0000", 2024-04-26T01:08:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1573000000000002e-05 2024-04-26T01:08:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304061+0000", 2024-04-26T01:08:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00083202500000000002 2024-04-26T01:08:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304318+0000", 2024-04-26T01:08:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025779299999999998 2024-04-26T01:08:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.306191+0000", 2024-04-26T01:08:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0018724010000000001 2024-04-26T01:08:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318520+0000", 2024-04-26T01:08:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012329028000000001 2024-04-26T01:08:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318628+0000", 2024-04-26T01:08:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00010778800000000001 2024-04-26T01:08:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318849+0000", 2024-04-26T01:08:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022091299999999999 2024-04-26T01:08:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:8 3.8s0 3:1b6b0095:::smithi16428812-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.303305+0000", 2024-04-26T01:08:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 36.852818550000002, 2024-04-26T01:08:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.020292667, 2024-04-26T01:08:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 8 2024-04-26T01:08:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303305+0000", 2024-04-26T01:08:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303305+0000", 2024-04-26T01:08:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303307+0000", 2024-04-26T01:08:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7770000000000001e-06 2024-04-26T01:08:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303320+0000", 2024-04-26T01:08:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3086e-05 2024-04-26T01:08:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303343+0000", 2024-04-26T01:08:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3323999999999999e-05 2024-04-26T01:08:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303387+0000", 2024-04-26T01:08:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3741e-05 2024-04-26T01:08:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.307388+0000", 2024-04-26T01:08:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0040010430000000001 2024-04-26T01:08:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.307645+0000", 2024-04-26T01:08:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025681400000000001 2024-04-26T01:08:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311893+0000", 2024-04-26T01:08:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.004247626 2024-04-26T01:08:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318857+0000", 2024-04-26T01:08:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0069646760000000004 2024-04-26T01:08:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323372+0000", 2024-04-26T01:08:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0045143329999999997 2024-04-26T01:08:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323598+0000", 2024-04-26T01:08:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000226247 2024-04-26T01:08:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:9 3.8s0 3:1b6b0095:::smithi16428812-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.303410+0000", 2024-04-26T01:08:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 36.852713469000001, 2024-04-26T01:08:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.028697769000000001, 2024-04-26T01:08:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 9 2024-04-26T01:08:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303410+0000", 2024-04-26T01:08:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303410+0000", 2024-04-26T01:08:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303412+0000", 2024-04-26T01:08:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.137e-06 2024-04-26T01:08:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303420+0000", 2024-04-26T01:08:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.3540000000000002e-06 2024-04-26T01:08:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303439+0000", 2024-04-26T01:08:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9227000000000001e-05 2024-04-26T01:08:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303455+0000", 2024-04-26T01:08:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5852999999999999e-05 2024-04-26T01:08:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312566+0000", 2024-04-26T01:08:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0091115209999999992 2024-04-26T01:08:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312793+0000", 2024-04-26T01:08:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022636999999999999 2024-04-26T01:08:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325814+0000", 2024-04-26T01:08:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013020893 2024-04-26T01:08:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326042+0000", 2024-04-26T01:08:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022811899999999999 2024-04-26T01:08:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.332108+0000", 2024-04-26T01:08:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0060662950000000002 2024-04-26T01:08:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:49 3.8s0 3:11b43273:::smithi16428812-17:head [append 0~278528,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.342928+0000", 2024-04-26T01:08:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 36.813195544000003, 2024-04-26T01:08:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0099803930000000006, 2024-04-26T01:08:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 49 2024-04-26T01:08:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.342928+0000", 2024-04-26T01:08:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.342928+0000", 2024-04-26T01:08:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.342931+0000", 2024-04-26T01:08:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.0000000000000001e-06 2024-04-26T01:08:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.343938+0000", 2024-04-26T01:08:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0010073129999999999 2024-04-26T01:08:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.343964+0000", 2024-04-26T01:08:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5936999999999999e-05 2024-04-26T01:08:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.343978+0000", 2024-04-26T01:08:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3481e-05 2024-04-26T01:08:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344123+0000", 2024-04-26T01:08:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014518199999999999 2024-04-26T01:08:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344393+0000", 2024-04-26T01:08:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00026982599999999997 2024-04-26T01:08:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.345686+0000", 2024-04-26T01:08:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001292774 2024-04-26T01:08:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.351912+0000", 2024-04-26T01:08:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.006225904 2024-04-26T01:08:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.352571+0000", 2024-04-26T01:08:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00065908000000000004 2024-04-26T01:08:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.352909+0000", 2024-04-26T01:08:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033789600000000002 2024-04-26T01:08:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:50 3.8s0 3:11b43273:::smithi16428812-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.344014+0000", 2024-04-26T01:08:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 36.812109769999999, 2024-04-26T01:08:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013017295999999999, 2024-04-26T01:08:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 50 2024-04-26T01:08:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344014+0000", 2024-04-26T01:08:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344014+0000", 2024-04-26T01:08:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344015+0000", 2024-04-26T01:08:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.437e-06 2024-04-26T01:08:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344022+0000", 2024-04-26T01:08:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.9970000000000001e-06 2024-04-26T01:08:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344034+0000", 2024-04-26T01:08:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1817000000000001e-05 2024-04-26T01:08:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344043+0000", 2024-04-26T01:08:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.8060000000000001e-06 2024-04-26T01:08:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.347851+0000", 2024-04-26T01:08:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0038083570000000001 2024-04-26T01:08:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.348142+0000", 2024-04-26T01:08:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00029042400000000001 2024-04-26T01:08:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350786+0000", 2024-04-26T01:08:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002644202 2024-04-26T01:08:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.352937+0000", 2024-04-26T01:08:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0021514910000000002 2024-04-26T01:08:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.356940+0000", 2024-04-26T01:08:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0040022740000000001 2024-04-26T01:08:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357031+0000", 2024-04-26T01:08:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.1490999999999995e-05 2024-04-26T01:08:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:51 3.8s0 3:11b43273:::smithi16428812-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.344056+0000", 2024-04-26T01:08:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 36.812067759000001, 2024-04-26T01:08:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.01557094, 2024-04-26T01:08:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 51 2024-04-26T01:08:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344056+0000", 2024-04-26T01:08:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344056+0000", 2024-04-26T01:08:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344057+0000", 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1699999999999997e-07 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344060+0000", 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.703e-06 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344071+0000", 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.057e-05 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344085+0000", 2024-04-26T01:08:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3997e-05 2024-04-26T01:08:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.351519+0000", 2024-04-26T01:08:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0074343079999999997 2024-04-26T01:08:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.351770+0000", 2024-04-26T01:08:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025095500000000003 2024-04-26T01:08:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357516+0000", 2024-04-26T01:08:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0057452919999999999 2024-04-26T01:08:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357612+0000", 2024-04-26T01:08:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6444e-05 2024-04-26T01:08:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.359627+0000", 2024-04-26T01:08:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002014854 2024-04-26T01:08:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 38/30 ECSubWrite(tid=85, reqid=osd.4.0:85, at_version=38'19, trim_to=0'0, roll_forward_to=38'15))", 2024-04-26T01:08:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.432265+0000", 2024-04-26T01:08:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 31.723858789000001, 2024-04-26T01:08:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011946251999999999, 2024-04-26T01:08:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432265+0000", 2024-04-26T01:08:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432265+0000", 2024-04-26T01:08:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432266+0000", 2024-04-26T01:08:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5900000000000005e-07 2024-04-26T01:08:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432295+0000", 2024-04-26T01:08:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9374999999999999e-05 2024-04-26T01:08:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441494+0000", 2024-04-26T01:08:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0091991369999999996 2024-04-26T01:08:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441529+0000", 2024-04-26T01:08:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.4677e-05 2024-04-26T01:08:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441702+0000", 2024-04-26T01:08:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017256500000000001 2024-04-26T01:08:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441731+0000", 2024-04-26T01:08:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8897000000000001e-05 2024-04-26T01:08:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.444168+0000", 2024-04-26T01:08:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0024374930000000002 2024-04-26T01:08:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.444211+0000", 2024-04-26T01:08:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3149000000000003e-05 2024-04-26T01:08:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 38/30 ECSubWrite(tid=87, reqid=client.4521.0:267, at_version=0'0, trim_to=0'0, roll_forward_to=38'21))", 2024-04-26T01:08:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.447653+0000", 2024-04-26T01:08:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 31.708470388999999, 2024-04-26T01:08:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.55546236199999999, 2024-04-26T01:08:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.447653+0000", 2024-04-26T01:08:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.447653+0000", 2024-04-26T01:08:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.447654+0000", 2024-04-26T01:08:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.8299999999999996e-07 2024-04-26T01:08:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.447665+0000", 2024-04-26T01:08:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0539e-05 2024-04-26T01:08:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.001792+0000", 2024-04-26T01:08:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.55412781 2024-04-26T01:08:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.001811+0000", 2024-04-26T01:08:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8236999999999998e-05 2024-04-26T01:08:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.001927+0000", 2024-04-26T01:08:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000116582 2024-04-26T01:08:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.001939+0000", 2024-04-26T01:08:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2312e-05 2024-04-26T01:08:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.003044+0000", 2024-04-26T01:08:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0011047819999999999 2024-04-26T01:08:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.003116+0000", 2024-04-26T01:08:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1416999999999997e-05 2024-04-26T01:08:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:288 3.8s0 3:1b6b0095:::smithi16428812-3:head [delete] snapc 4=[2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.532919+0000", 2024-04-26T01:08:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 31.623204205, 2024-04-26T01:08:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0091303889999999992, 2024-04-26T01:08:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 288 2024-04-26T01:08:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532919+0000", 2024-04-26T01:08:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532919+0000", 2024-04-26T01:08:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.205 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.205 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.205 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.205 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532921+0000", 2024-04-26T01:08:36.205 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8810000000000001e-06 2024-04-26T01:08:36.205 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.205 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.205 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.205 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532929+0000", 2024-04-26T01:08:36.205 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8599999999999993e-06 2024-04-26T01:08:36.205 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.206 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.206 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.206 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532950+0000", 2024-04-26T01:08:36.206 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0738999999999999e-05 2024-04-26T01:08:36.206 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.206 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.206 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.206 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.532960+0000", 2024-04-26T01:08:36.206 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5990000000000008e-06 2024-04-26T01:08:36.206 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.206 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.206 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.207 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.533062+0000", 2024-04-26T01:08:36.207 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000102402 2024-04-26T01:08:36.207 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.207 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.207 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:36.207 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.533241+0000", 2024-04-26T01:08:36.207 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017890899999999999 2024-04-26T01:08:36.207 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.207 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.207 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.207 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.534156+0000", 2024-04-26T01:08:36.207 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00091496199999999996 2024-04-26T01:08:36.208 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.208 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.208 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.208 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.535119+0000", 2024-04-26T01:08:36.208 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00096345000000000003 2024-04-26T01:08:36.208 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.208 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.208 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:36.208 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.541980+0000", 2024-04-26T01:08:36.208 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0068602719999999997 2024-04-26T01:08:36.208 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.208 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.209 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.209 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.542050+0000", 2024-04-26T01:08:36.209 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.0314999999999993e-05 2024-04-26T01:08:36.209 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.209 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.209 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.209 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.209 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.209 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:290 3.8s0 3:11b43273:::smithi16428812-17:head [read 0~1294336,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:36.209 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.545945+0000", 2024-04-26T01:08:36.209 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 31.610178796, 2024-04-26T01:08:36.210 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0074298849999999998, 2024-04-26T01:08:36.210 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.210 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.210 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:36.210 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:36.210 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:36.210 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:36.210 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 290 2024-04-26T01:08:36.210 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.210 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.210 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.210 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.210 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545945+0000", 2024-04-26T01:08:36.211 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.211 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.211 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.211 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545945+0000", 2024-04-26T01:08:36.211 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.211 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.211 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.211 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545946+0000", 2024-04-26T01:08:36.211 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.0500000000000003e-07 2024-04-26T01:08:36.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.212 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.212 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.212 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545951+0000", 2024-04-26T01:08:36.212 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0429999999999998e-06 2024-04-26T01:08:36.212 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.212 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.212 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.212 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545960+0000", 2024-04-26T01:08:36.212 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.3859999999999996e-06 2024-04-26T01:08:36.212 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.212 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.212 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.213 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.545972+0000", 2024-04-26T01:08:36.213 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2234e-05 2024-04-26T01:08:36.213 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.213 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.213 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.213 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.547938+0000", 2024-04-26T01:08:36.213 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001965989 2024-04-26T01:08:36.213 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.213 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.213 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:36.213 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.548007+0000", 2024-04-26T01:08:36.213 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.9151000000000003e-05 2024-04-26T01:08:36.213 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.214 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.214 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.214 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.553375+0000", 2024-04-26T01:08:36.214 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0053673770000000004 2024-04-26T01:08:36.214 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.214 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.214 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.214 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.214 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.214 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.cs6 50/44 ECSubWrite(tid=22, reqid=osd.9.0:22, at_version=50'37, trim_to=0'0, roll_forward_to=47'35))", 2024-04-26T01:08:36.214 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.057328+0000", 2024-04-26T01:08:36.215 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.098796037, 2024-04-26T01:08:36.215 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010531041999999999, 2024-04-26T01:08:36.215 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.215 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.215 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.215 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.215 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.215 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.215 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057328+0000", 2024-04-26T01:08:36.215 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.215 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.215 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.216 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.216 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057328+0000", 2024-04-26T01:08:36.216 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.216 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.216 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.216 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.216 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057328+0000", 2024-04-26T01:08:36.216 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.4700000000000001e-07 2024-04-26T01:08:36.216 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.216 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.216 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.216 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057387+0000", 2024-04-26T01:08:36.217 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.8983000000000001e-05 2024-04-26T01:08:36.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.217 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.217 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.217 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057937+0000", 2024-04-26T01:08:36.217 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00054932600000000005 2024-04-26T01:08:36.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.217 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.217 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.217 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057963+0000", 2024-04-26T01:08:36.217 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6794000000000001e-05 2024-04-26T01:08:36.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.218 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.218 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.218 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058163+0000", 2024-04-26T01:08:36.218 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000199834 2024-04-26T01:08:36.218 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.218 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.218 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.218 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058192+0000", 2024-04-26T01:08:36.218 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.921e-05 2024-04-26T01:08:36.218 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.218 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.218 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.219 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.067751+0000", 2024-04-26T01:08:36.219 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0095586939999999995 2024-04-26T01:08:36.219 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.219 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.219 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.219 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.067859+0000", 2024-04-26T01:08:36.219 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00010755399999999999 2024-04-26T01:08:36.219 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.219 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.219 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.219 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.220 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.220 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/44 ECSubWrite(tid=215, reqid=client.4521.0:490, at_version=50'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-26T01:08:36.220 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.059123+0000", 2024-04-26T01:08:36.220 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.097000476000002, 2024-04-26T01:08:36.220 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.014980383999999999, 2024-04-26T01:08:36.220 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.220 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.220 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.220 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.220 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.220 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.220 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.059123+0000", 2024-04-26T01:08:36.221 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.221 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.221 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.221 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.059123+0000", 2024-04-26T01:08:36.221 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.221 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.221 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.221 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.059124+0000", 2024-04-26T01:08:36.221 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0839999999999999e-06 2024-04-26T01:08:36.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.222 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.222 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.222 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.059964+0000", 2024-04-26T01:08:36.222 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00083960600000000001 2024-04-26T01:08:36.222 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.222 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.222 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.222 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066110+0000", 2024-04-26T01:08:36.222 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0061464900000000001 2024-04-26T01:08:36.222 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.223 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.223 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.223 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066137+0000", 2024-04-26T01:08:36.223 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6307000000000001e-05 2024-04-26T01:08:36.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.223 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.223 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.223 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066336+0000", 2024-04-26T01:08:36.223 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019909100000000001 2024-04-26T01:08:36.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.223 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.223 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.223 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066365+0000", 2024-04-26T01:08:36.224 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9490000000000001e-05 2024-04-26T01:08:36.224 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.224 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.224 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.224 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.073999+0000", 2024-04-26T01:08:36.224 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0076337089999999998 2024-04-26T01:08:36.224 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.224 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.224 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.224 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.074104+0000", 2024-04-26T01:08:36.224 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000104607 2024-04-26T01:08:36.224 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.224 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.225 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.225 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.225 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.225 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 50/30 ECSubWriteReply(tid=41, last_complete=50'26, committed=1, applied=1))", 2024-04-26T01:08:36.225 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.060433+0000", 2024-04-26T01:08:36.225 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.095691006999999, 2024-04-26T01:08:36.225 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0074890549999999997, 2024-04-26T01:08:36.225 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.225 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.225 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.225 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.225 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.225 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.225 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060433+0000", 2024-04-26T01:08:36.226 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.226 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.226 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.226 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.226 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060433+0000", 2024-04-26T01:08:36.226 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.226 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.226 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.226 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.226 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060433+0000", 2024-04-26T01:08:36.226 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.5000000000000002e-07 2024-04-26T01:08:36.226 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.227 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.227 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.227 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060441+0000", 2024-04-26T01:08:36.227 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.233e-06 2024-04-26T01:08:36.227 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.227 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.227 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.227 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.062232+0000", 2024-04-26T01:08:36.227 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001791441 2024-04-26T01:08:36.227 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.227 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.227 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.228 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.062254+0000", 2024-04-26T01:08:36.228 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1846999999999999e-05 2024-04-26T01:08:36.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.228 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.228 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.228 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066505+0000", 2024-04-26T01:08:36.228 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0042505879999999996 2024-04-26T01:08:36.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.228 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.228 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.228 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.067922+0000", 2024-04-26T01:08:36.228 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0014171959999999999 2024-04-26T01:08:36.228 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.229 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.229 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.229 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.229 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/44 ECSubWrite(tid=218, reqid=osd.4.0:218, at_version=50'34, trim_to=0'0, roll_forward_to=47'30))", 2024-04-26T01:08:36.229 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.065908+0000", 2024-04-26T01:08:36.229 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.090215279999999, 2024-04-26T01:08:36.229 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010766286, 2024-04-26T01:08:36.229 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.229 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.229 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.229 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.229 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.230 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.230 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065908+0000", 2024-04-26T01:08:36.230 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.230 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.230 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.230 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.230 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065908+0000", 2024-04-26T01:08:36.230 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.230 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.230 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.230 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.230 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065911+0000", 2024-04-26T01:08:36.231 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5900000000000002e-06 2024-04-26T01:08:36.231 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.231 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.231 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.231 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065971+0000", 2024-04-26T01:08:36.231 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.0152999999999998e-05 2024-04-26T01:08:36.231 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.231 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.231 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.231 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066224+0000", 2024-04-26T01:08:36.231 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025274800000000001 2024-04-26T01:08:36.231 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.232 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.232 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.232 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066242+0000", 2024-04-26T01:08:36.232 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8499999999999999e-05 2024-04-26T01:08:36.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.232 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.232 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.232 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.068803+0000", 2024-04-26T01:08:36.232 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002560414 2024-04-26T01:08:36.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.232 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.232 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.232 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.068831+0000", 2024-04-26T01:08:36.233 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8586999999999999e-05 2024-04-26T01:08:36.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.233 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.233 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.233 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076582+0000", 2024-04-26T01:08:36.233 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0077506160000000001 2024-04-26T01:08:36.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.233 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.233 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.233 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076675+0000", 2024-04-26T01:08:36.233 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2677999999999996e-05 2024-04-26T01:08:36.233 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.234 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.234 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.234 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.234 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.234 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 50/44 ECSubWrite(tid=219, reqid=osd.4.0:219, at_version=50'58, trim_to=0'0, roll_forward_to=47'56))", 2024-04-26T01:08:36.234 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.065984+0000", 2024-04-26T01:08:36.234 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.090140056999999, 2024-04-26T01:08:36.234 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.018105768000000001, 2024-04-26T01:08:36.234 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.234 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.234 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.234 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.234 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.235 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.235 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065984+0000", 2024-04-26T01:08:36.235 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.235 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.235 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.235 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065984+0000", 2024-04-26T01:08:36.235 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.235 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.235 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.235 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065984+0000", 2024-04-26T01:08:36.236 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.7700000000000004e-07 2024-04-26T01:08:36.236 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.236 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.236 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.236 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066035+0000", 2024-04-26T01:08:36.236 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0470000000000003e-05 2024-04-26T01:08:36.236 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.236 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.236 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.236 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077389+0000", 2024-04-26T01:08:36.236 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011354326 2024-04-26T01:08:36.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.237 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.237 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.237 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077419+0000", 2024-04-26T01:08:36.237 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.0006e-05 2024-04-26T01:08:36.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.237 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.237 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.237 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077629+0000", 2024-04-26T01:08:36.237 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000209471 2024-04-26T01:08:36.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.237 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.237 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.237 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077685+0000", 2024-04-26T01:08:36.238 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.6674e-05 2024-04-26T01:08:36.238 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.238 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.238 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.238 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.083973+0000", 2024-04-26T01:08:36.238 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0062876870000000001 2024-04-26T01:08:36.238 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.238 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.238 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.238 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.084089+0000", 2024-04-26T01:08:36.238 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000116357 2024-04-26T01:08:36.238 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.239 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.239 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.239 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.239 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/30 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=50'32, trim_to=0'0, roll_forward_to=45'30))", 2024-04-26T01:08:36.239 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.066181+0000", 2024-04-26T01:08:36.239 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.089942636, 2024-04-26T01:08:36.239 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011395499, 2024-04-26T01:08:36.239 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.239 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.239 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.239 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.239 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.240 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.240 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066181+0000", 2024-04-26T01:08:36.240 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.240 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.240 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.240 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066181+0000", 2024-04-26T01:08:36.240 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.240 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.240 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.240 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066183+0000", 2024-04-26T01:08:36.241 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.767e-06 2024-04-26T01:08:36.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.241 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.241 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.241 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066240+0000", 2024-04-26T01:08:36.241 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.7099999999999999e-05 2024-04-26T01:08:36.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.241 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.241 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.241 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066338+0000", 2024-04-26T01:08:36.241 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7738000000000005e-05 2024-04-26T01:08:36.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.241 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.242 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.242 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066361+0000", 2024-04-26T01:08:36.242 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3615999999999999e-05 2024-04-26T01:08:36.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.242 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.242 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.070266+0000", 2024-04-26T01:08:36.242 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0039045659999999999 2024-04-26T01:08:36.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.242 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.243 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.070819+0000", 2024-04-26T01:08:36.243 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00055286599999999995 2024-04-26T01:08:36.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.243 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.243 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.243 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077394+0000", 2024-04-26T01:08:36.243 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0065747990000000001 2024-04-26T01:08:36.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.243 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.243 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.243 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077577+0000", 2024-04-26T01:08:36.243 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00018304700000000001 2024-04-26T01:08:36.243 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.243 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.244 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.244 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.244 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.244 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s5 58/44 ECSubWrite(tid=292, reqid=client.4521.0:584, at_version=58'51, trim_to=0'0, roll_forward_to=56'49))", 2024-04-26T01:08:36.244 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:24.169649+0000", 2024-04-26T01:08:36.244 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 11.986474441, 2024-04-26T01:08:36.244 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0073139779999999996, 2024-04-26T01:08:36.244 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.244 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.244 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.244 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.244 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.244 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.245 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.169649+0000", 2024-04-26T01:08:36.245 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.245 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.245 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.245 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.245 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.169649+0000", 2024-04-26T01:08:36.245 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.245 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.245 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.245 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.245 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.169652+0000", 2024-04-26T01:08:36.245 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4499999999999998e-06 2024-04-26T01:08:36.245 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.246 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.246 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.246 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.172490+0000", 2024-04-26T01:08:36.246 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0028385870000000001 2024-04-26T01:08:36.246 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.246 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.246 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.246 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.172586+0000", 2024-04-26T01:08:36.246 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5303999999999994e-05 2024-04-26T01:08:36.246 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.246 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.246 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.247 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.172615+0000", 2024-04-26T01:08:36.247 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9527e-05 2024-04-26T01:08:36.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.247 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.247 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.247 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.172782+0000", 2024-04-26T01:08:36.247 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016702700000000001 2024-04-26T01:08:36.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.247 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.247 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.247 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.172809+0000", 2024-04-26T01:08:36.247 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7140000000000001e-05 2024-04-26T01:08:36.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.248 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.248 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.248 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.176865+0000", 2024-04-26T01:08:36.248 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0040557589999999999 2024-04-26T01:08:36.248 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.248 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.248 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.248 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.176963+0000", 2024-04-26T01:08:36.248 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.8183999999999994e-05 2024-04-26T01:08:36.248 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.248 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.248 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.249 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.249 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.249 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.cs6 59/44 ECSubWrite(tid=36, reqid=osd.9.0:36, at_version=59'49, trim_to=0'0, roll_forward_to=57'47))", 2024-04-26T01:08:36.249 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:25.171981+0000", 2024-04-26T01:08:36.249 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 10.984142727, 2024-04-26T01:08:36.249 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0078630269999999999, 2024-04-26T01:08:36.249 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.249 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.249 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.249 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.249 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.249 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.249 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.171981+0000", 2024-04-26T01:08:36.250 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.250 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.250 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.250 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.171981+0000", 2024-04-26T01:08:36.250 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.250 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.250 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.250 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.171982+0000", 2024-04-26T01:08:36.250 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.9400000000000004e-07 2024-04-26T01:08:36.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.251 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.251 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.251 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.172040+0000", 2024-04-26T01:08:36.251 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.8282999999999997e-05 2024-04-26T01:08:36.251 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.251 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.251 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.251 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.173724+0000", 2024-04-26T01:08:36.251 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001683429 2024-04-26T01:08:36.251 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.251 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.252 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.252 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.173746+0000", 2024-04-26T01:08:36.252 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2572999999999999e-05 2024-04-26T01:08:36.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.252 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.252 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.252 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.173962+0000", 2024-04-26T01:08:36.252 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00021617000000000001 2024-04-26T01:08:36.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.252 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.252 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.253 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.173990+0000", 2024-04-26T01:08:36.253 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7889999999999999e-05 2024-04-26T01:08:36.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.253 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.253 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.179750+0000", 2024-04-26T01:08:36.253 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0057598950000000001 2024-04-26T01:08:36.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.253 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.253 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.179844+0000", 2024-04-26T01:08:36.253 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.3893000000000004e-05 2024-04-26T01:08:36.254 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.254 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.254 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.254 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.254 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s5 59/44 ECSubWrite(tid=326, reqid=osd.4.0:326, at_version=59'60, trim_to=0'0, roll_forward_to=58'58))", 2024-04-26T01:08:36.254 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:25.178499+0000", 2024-04-26T01:08:36.254 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 10.977624212, 2024-04-26T01:08:36.254 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0078318299999999997, 2024-04-26T01:08:36.254 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.254 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.254 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.254 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.255 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.255 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.178499+0000", 2024-04-26T01:08:36.255 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.255 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.255 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.178499+0000", 2024-04-26T01:08:36.255 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.255 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.256 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.178500+0000", 2024-04-26T01:08:36.256 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.4000000000000001e-07 2024-04-26T01:08:36.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.256 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.256 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.256 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.178554+0000", 2024-04-26T01:08:36.256 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.4113999999999997e-05 2024-04-26T01:08:36.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.256 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.256 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.256 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.178656+0000", 2024-04-26T01:08:36.256 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00010205 2024-04-26T01:08:36.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.257 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.257 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.178680+0000", 2024-04-26T01:08:36.257 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3297000000000001e-05 2024-04-26T01:08:36.257 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.257 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.257 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.178845+0000", 2024-04-26T01:08:36.257 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016512699999999999 2024-04-26T01:08:36.257 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.257 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.258 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.178868+0000", 2024-04-26T01:08:36.258 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.338e-05 2024-04-26T01:08:36.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.258 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.258 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.258 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.186249+0000", 2024-04-26T01:08:36.258 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0073805850000000003 2024-04-26T01:08:36.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.258 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.258 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.258 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.186331+0000", 2024-04-26T01:08:36.258 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.2637000000000002e-05 2024-04-26T01:08:36.259 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.259 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.259 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.259 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.259 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s4 60/30 ECSubWrite(tid=70, reqid=osd.14.0:70, at_version=60'57, trim_to=0'0, roll_forward_to=59'53))", 2024-04-26T01:08:36.259 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:26.205578+0000", 2024-04-26T01:08:36.259 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.9505459330000008, 2024-04-26T01:08:36.259 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0070530890000000002, 2024-04-26T01:08:36.259 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.259 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.259 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.259 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.260 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.260 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.205578+0000", 2024-04-26T01:08:36.260 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.260 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.260 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.205578+0000", 2024-04-26T01:08:36.260 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.260 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.260 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.205580+0000", 2024-04-26T01:08:36.261 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8169999999999999e-06 2024-04-26T01:08:36.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.261 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.261 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.205630+0000", 2024-04-26T01:08:36.261 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0492999999999999e-05 2024-04-26T01:08:36.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.261 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.261 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.205702+0000", 2024-04-26T01:08:36.261 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.2193999999999994e-05 2024-04-26T01:08:36.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.262 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.262 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.205717+0000", 2024-04-26T01:08:36.262 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4419999999999999e-05 2024-04-26T01:08:36.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.262 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.262 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.262 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.205868+0000", 2024-04-26T01:08:36.262 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015085700000000001 2024-04-26T01:08:36.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.262 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.262 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.262 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.205894+0000", 2024-04-26T01:08:36.263 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6571e-05 2024-04-26T01:08:36.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.263 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.263 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.263 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.212515+0000", 2024-04-26T01:08:36.263 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0066213139999999997 2024-04-26T01:08:36.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.263 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.263 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.263 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.212631+0000", 2024-04-26T01:08:36.263 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000115423 2024-04-26T01:08:36.263 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.263 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.264 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.264 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.264 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.264 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:36.265 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:36.260+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:36.266 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:08:36.366 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:36.363+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:36.440 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:36.440 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:36.440 INFO:teuthology.orchestra.run.smithi012.stdout: "num_ops": 0 2024-04-26T01:08:36.440 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:36.454 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:08:36.468 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:36.467+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:36.567 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:36.567+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:36.575 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:36.575 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:36.575 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:08:36.575 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:08:36.575 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:36.590 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:08:36.650 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:36.668 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:36.664+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:36.698 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:36.698 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:36.721 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:08:36.721 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:08:36.721 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:08:36.721 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.721 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/30 ECSubWrite(tid=9, reqid=client.4521.0:38, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:36.721 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.327957+0000", 2024-04-26T01:08:36.721 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.391395236000001, 2024-04-26T01:08:36.721 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.029348438000000001, 2024-04-26T01:08:36.721 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.721 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.721 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.722 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.722 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.722 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.722 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327957+0000", 2024-04-26T01:08:36.722 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.722 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.722 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.722 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.722 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327957+0000", 2024-04-26T01:08:36.722 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.722 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.722 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.722 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327958+0000", 2024-04-26T01:08:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.6300000000000005e-07 2024-04-26T01:08:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327988+0000", 2024-04-26T01:08:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.0577000000000002e-05 2024-04-26T01:08:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.354381+0000", 2024-04-26T01:08:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.026392280000000001 2024-04-26T01:08:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.724 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.354407+0000", 2024-04-26T01:08:36.724 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6783000000000001e-05 2024-04-26T01:08:36.724 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.724 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.724 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.724 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.354542+0000", 2024-04-26T01:08:36.724 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013464299999999999 2024-04-26T01:08:36.724 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.724 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.724 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.724 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.354563+0000", 2024-04-26T01:08:36.724 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0744999999999999e-05 2024-04-26T01:08:36.724 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.724 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.725 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.725 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357241+0000", 2024-04-26T01:08:36.725 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0026777929999999999 2024-04-26T01:08:36.725 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.725 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.725 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.725 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357306+0000", 2024-04-26T01:08:36.725 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.5054000000000004e-05 2024-04-26T01:08:36.725 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.725 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.725 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.725 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.725 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.726 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpRead(3.1s2 33/30 ECSubRead(tid=30, to_read={3:88871701:::smithi16428812-34:head=0,4096,0}, subchunks={3:88871701:::smithi16428812-34:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:36.726 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.420990+0000", 2024-04-26T01:08:36.726 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.298362580999999, 2024-04-26T01:08:36.726 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.028504367999999999, 2024-04-26T01:08:36.726 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.726 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.726 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.726 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.726 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.726 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.726 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.420990+0000", 2024-04-26T01:08:36.727 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.727 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.727 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.727 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.727 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.420990+0000", 2024-04-26T01:08:36.727 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.727 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.727 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.727 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.727 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.420991+0000", 2024-04-26T01:08:36.727 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.254e-06 2024-04-26T01:08:36.727 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.727 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.727 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.728 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.421000+0000", 2024-04-26T01:08:36.728 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.8370000000000003e-06 2024-04-26T01:08:36.728 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.728 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.728 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.728 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.448556+0000", 2024-04-26T01:08:36.728 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.027556243000000001 2024-04-26T01:08:36.728 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.728 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.728 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.728 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.448583+0000", 2024-04-26T01:08:36.728 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6761e-05 2024-04-26T01:08:36.728 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.728 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.728 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.729 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.449059+0000", 2024-04-26T01:08:36.729 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00047558499999999999 2024-04-26T01:08:36.729 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.729 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.729 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.729 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.449494+0000", 2024-04-26T01:08:36.729 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00043568799999999999 2024-04-26T01:08:36.729 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.729 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.730 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.730 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.730 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.730 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:487 3.cs0 3:3bb987cc:::smithi16428812-47:head [read 0~1884160,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-04-26T01:08:36.730 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.037904+0000", 2024-04-26T01:08:36.730 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.681448500999998, 2024-04-26T01:08:36.730 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.044199702, 2024-04-26T01:08:36.730 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.730 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.730 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:36.730 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:36.730 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:36.731 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:36.731 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 487 2024-04-26T01:08:36.731 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.731 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.731 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.731 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.731 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037904+0000", 2024-04-26T01:08:36.731 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.731 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.731 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.731 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.731 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037904+0000", 2024-04-26T01:08:36.732 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.732 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.732 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.732 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.732 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037908+0000", 2024-04-26T01:08:36.732 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.1829999999999996e-06 2024-04-26T01:08:36.732 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.732 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.732 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.732 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037926+0000", 2024-04-26T01:08:36.732 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7859999999999998e-05 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037974+0000", 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.8494e-05 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037999+0000", 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4899999999999999e-05 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038165+0000", 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016607699999999999 2024-04-26T01:08:36.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.734 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.734 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op must wait for map", 2024-04-26T01:08:36.734 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038179+0000", 2024-04-26T01:08:36.734 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.403e-05 2024-04-26T01:08:36.734 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.734 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.734 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.734 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.049988+0000", 2024-04-26T01:08:36.734 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011808206999999999 2024-04-26T01:08:36.734 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.734 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.734 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:36.735 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.050141+0000", 2024-04-26T01:08:36.735 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015328799999999999 2024-04-26T01:08:36.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.735 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.735 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.735 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.082104+0000", 2024-04-26T01:08:36.735 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.031962663000000002 2024-04-26T01:08:36.735 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.735 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.735 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.735 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.735 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:488 3.cs0 3:3bb987cc:::smithi16428812-47:head [read 0~1884160,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-04-26T01:08:36.735 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.038063+0000", 2024-04-26T01:08:36.736 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.681289936999999, 2024-04-26T01:08:36.736 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.047336889, 2024-04-26T01:08:36.736 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.736 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.736 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:36.736 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:36.736 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:36.736 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:36.736 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 488 2024-04-26T01:08:36.736 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.736 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.736 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.736 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.736 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038063+0000", 2024-04-26T01:08:36.737 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.737 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.737 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.737 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.737 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038063+0000", 2024-04-26T01:08:36.737 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.737 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.737 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.737 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.737 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038064+0000", 2024-04-26T01:08:36.737 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3799999999999999e-06 2024-04-26T01:08:36.737 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.737 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.737 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.737 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038075+0000", 2024-04-26T01:08:36.738 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0993e-05 2024-04-26T01:08:36.738 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.738 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.738 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.738 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038096+0000", 2024-04-26T01:08:36.738 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1284000000000002e-05 2024-04-26T01:08:36.738 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.738 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.738 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.738 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038112+0000", 2024-04-26T01:08:36.738 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5577e-05 2024-04-26T01:08:36.738 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.738 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.738 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.739 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038366+0000", 2024-04-26T01:08:36.739 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025445099999999999 2024-04-26T01:08:36.739 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.739 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.739 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:36.739 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038390+0000", 2024-04-26T01:08:36.739 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3782999999999999e-05 2024-04-26T01:08:36.739 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.739 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.739 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.739 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051105+0000", 2024-04-26T01:08:36.740 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012714757 2024-04-26T01:08:36.740 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.740 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.740 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:36.740 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051362+0000", 2024-04-26T01:08:36.740 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025746800000000001 2024-04-26T01:08:36.740 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.740 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.740 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.740 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.085399+0000", 2024-04-26T01:08:36.741 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.034037195999999999 2024-04-26T01:08:36.741 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.741 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.741 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.741 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.741 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.741 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:489 3.cs0 3:3bb987cc:::smithi16428812-47:head [read 0~1884160,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-04-26T01:08:36.741 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.038135+0000", 2024-04-26T01:08:36.741 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.681217827000001, 2024-04-26T01:08:36.741 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.049787337000000001, 2024-04-26T01:08:36.741 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.741 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.741 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:36.741 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:36.741 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:36.742 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:36.742 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 489 2024-04-26T01:08:36.742 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.742 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.742 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.742 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.742 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038135+0000", 2024-04-26T01:08:36.742 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.742 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.742 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.742 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.742 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038135+0000", 2024-04-26T01:08:36.742 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.742 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.743 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.743 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.743 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038136+0000", 2024-04-26T01:08:36.743 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1739999999999999e-06 2024-04-26T01:08:36.743 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.743 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.743 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.743 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038144+0000", 2024-04-26T01:08:36.743 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.746e-06 2024-04-26T01:08:36.743 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.743 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.743 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.744 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038167+0000", 2024-04-26T01:08:36.744 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2977e-05 2024-04-26T01:08:36.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.744 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.744 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.744 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038182+0000", 2024-04-26T01:08:36.744 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5543e-05 2024-04-26T01:08:36.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.744 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.744 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.746 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038497+0000", 2024-04-26T01:08:36.746 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00031515200000000002 2024-04-26T01:08:36.746 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.746 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.747 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:36.747 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038521+0000", 2024-04-26T01:08:36.747 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3706999999999999e-05 2024-04-26T01:08:36.747 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.747 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.747 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.747 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052089+0000", 2024-04-26T01:08:36.747 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013568347 2024-04-26T01:08:36.747 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.747 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.747 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:36.747 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052357+0000", 2024-04-26T01:08:36.747 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00026783200000000002 2024-04-26T01:08:36.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.748 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.748 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.748 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.087922+0000", 2024-04-26T01:08:36.748 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.035564858999999997 2024-04-26T01:08:36.748 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.748 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.748 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.748 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.748 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpRead(3.cs0 50/44 ECSubRead(tid=19, to_read={3:3bb987cc:::smithi16428812-47:head=0,471040,0}, subchunks={3:3bb987cc:::smithi16428812-47:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:36.748 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.050864+0000", 2024-04-26T01:08:36.748 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.668488819, 2024-04-26T01:08:36.748 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.023549771000000001, 2024-04-26T01:08:36.749 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.749 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.749 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.749 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.749 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.749 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.749 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.050864+0000", 2024-04-26T01:08:36.749 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.749 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.749 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.749 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.749 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.050864+0000", 2024-04-26T01:08:36.749 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.749 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.749 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.750 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.750 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.050864+0000", 2024-04-26T01:08:36.750 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.750 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.750 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.750 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.750 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.050864+0000", 2024-04-26T01:08:36.750 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.750 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.750 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.750 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.750 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053143+0000", 2024-04-26T01:08:36.750 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0022794400000000002 2024-04-26T01:08:36.750 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.750 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.751 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.751 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053173+0000", 2024-04-26T01:08:36.751 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9635999999999999e-05 2024-04-26T01:08:36.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.751 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.751 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.751 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.069570+0000", 2024-04-26T01:08:36.751 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.016397047000000001 2024-04-26T01:08:36.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.751 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.751 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.751 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.074413+0000", 2024-04-26T01:08:36.751 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0048436479999999999 2024-04-26T01:08:36.751 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.752 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.752 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.752 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.752 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpRead(3.cs0 50/44 ECSubRead(tid=20, to_read={3:3bb987cc:::smithi16428812-47:head=0,471040,0}, subchunks={3:3bb987cc:::smithi16428812-47:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:36.752 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.051911+0000", 2024-04-26T01:08:36.752 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.667440971000001, 2024-04-26T01:08:36.752 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.023435303000000001, 2024-04-26T01:08:36.752 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.752 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.752 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.752 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.752 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.752 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.753 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051911+0000", 2024-04-26T01:08:36.753 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.753 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.753 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051911+0000", 2024-04-26T01:08:36.753 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.753 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.753 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051911+0000", 2024-04-26T01:08:36.753 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.753 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.754 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051911+0000", 2024-04-26T01:08:36.754 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.754 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.754 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.754 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053257+0000", 2024-04-26T01:08:36.754 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001345759 2024-04-26T01:08:36.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.754 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.754 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.754 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053301+0000", 2024-04-26T01:08:36.754 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3482999999999997e-05 2024-04-26T01:08:36.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.754 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.755 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.755 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075035+0000", 2024-04-26T01:08:36.755 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.021734690000000001 2024-04-26T01:08:36.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.755 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.755 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.755 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075347+0000", 2024-04-26T01:08:36.755 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00031137100000000001 2024-04-26T01:08:36.755 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.755 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.755 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.755 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.755 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpRead(3.cs0 50/44 ECSubRead(tid=21, to_read={3:3bb987cc:::smithi16428812-47:head=0,471040,0}, subchunks={3:3bb987cc:::smithi16428812-47:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:36.756 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.052958+0000", 2024-04-26T01:08:36.756 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.66639447, 2024-04-26T01:08:36.756 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.023405313000000001, 2024-04-26T01:08:36.756 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.756 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.756 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.756 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.756 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.756 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.756 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052958+0000", 2024-04-26T01:08:36.756 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.756 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.756 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.756 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052958+0000", 2024-04-26T01:08:36.757 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.757 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.757 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.757 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.757 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052958+0000", 2024-04-26T01:08:36.757 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.757 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.757 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.757 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.757 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052958+0000", 2024-04-26T01:08:36.757 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.757 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.757 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.757 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.757 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053516+0000", 2024-04-26T01:08:36.758 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00055782599999999998 2024-04-26T01:08:36.758 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.758 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.758 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.758 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053561+0000", 2024-04-26T01:08:36.758 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.4777000000000002e-05 2024-04-26T01:08:36.758 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.758 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.758 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.758 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075975+0000", 2024-04-26T01:08:36.758 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.022413968999999999 2024-04-26T01:08:36.758 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.758 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.758 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.759 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076363+0000", 2024-04-26T01:08:36.759 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00038874100000000001 2024-04-26T01:08:36.759 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.759 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.759 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.759 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.759 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.759 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=19, attrs_read=0))", 2024-04-26T01:08:36.759 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.053888+0000", 2024-04-26T01:08:36.759 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.665464143000001, 2024-04-26T01:08:36.759 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.021805537, 2024-04-26T01:08:36.759 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.759 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.759 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.759 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.760 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.760 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.760 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053888+0000", 2024-04-26T01:08:36.760 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.760 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.760 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.760 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.760 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053888+0000", 2024-04-26T01:08:36.760 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.760 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.760 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.760 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.760 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053890+0000", 2024-04-26T01:08:36.760 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9870000000000002e-06 2024-04-26T01:08:36.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.761 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.761 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.761 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055110+0000", 2024-04-26T01:08:36.761 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0012193379999999999 2024-04-26T01:08:36.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.761 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.761 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.761 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055613+0000", 2024-04-26T01:08:36.761 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00050335600000000005 2024-04-26T01:08:36.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.761 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.761 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.761 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055632+0000", 2024-04-26T01:08:36.761 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.855e-05 2024-04-26T01:08:36.762 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.762 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.762 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.762 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075594+0000", 2024-04-26T01:08:36.762 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.019962005000000001 2024-04-26T01:08:36.762 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.762 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.762 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.762 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075694+0000", 2024-04-26T01:08:36.762 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000100301 2024-04-26T01:08:36.762 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.762 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.762 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.762 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.762 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.763 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:08:36.763 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.055120+0000", 2024-04-26T01:08:36.763 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.664232891000001, 2024-04-26T01:08:36.763 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.023938912999999999, 2024-04-26T01:08:36.763 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.763 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.763 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.763 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.763 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.763 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.763 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055120+0000", 2024-04-26T01:08:36.763 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.763 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.763 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.764 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.764 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055120+0000", 2024-04-26T01:08:36.764 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.764 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.764 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.764 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.764 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055121+0000", 2024-04-26T01:08:36.764 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.367e-06 2024-04-26T01:08:36.764 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.764 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.764 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.764 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055787+0000", 2024-04-26T01:08:36.764 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000665702 2024-04-26T01:08:36.764 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.764 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.765 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.765 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063149+0000", 2024-04-26T01:08:36.765 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0073619590000000004 2024-04-26T01:08:36.765 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.765 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.765 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.765 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063177+0000", 2024-04-26T01:08:36.765 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8592999999999999e-05 2024-04-26T01:08:36.765 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.765 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.765 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.765 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.078938+0000", 2024-04-26T01:08:36.765 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.015760687999999998 2024-04-26T01:08:36.765 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.765 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.766 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.766 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.079058+0000", 2024-04-26T01:08:36.766 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000120604 2024-04-26T01:08:36.766 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.766 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.766 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.766 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.766 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.766 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 48/44 ECSubReadReply(tid=19, attrs_read=0))", 2024-04-26T01:08:36.766 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.055536+0000", 2024-04-26T01:08:36.766 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.663816176000001, 2024-04-26T01:08:36.766 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.019985012999999999, 2024-04-26T01:08:36.767 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.767 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.767 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.767 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.767 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.767 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.767 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055536+0000", 2024-04-26T01:08:36.767 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.767 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.767 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.767 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.767 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055536+0000", 2024-04-26T01:08:36.767 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.767 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.768 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.768 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.768 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055538+0000", 2024-04-26T01:08:36.768 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5969999999999999e-06 2024-04-26T01:08:36.768 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.768 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.768 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.768 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057540+0000", 2024-04-26T01:08:36.768 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0020016449999999998 2024-04-26T01:08:36.768 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.768 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.768 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.768 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063281+0000", 2024-04-26T01:08:36.768 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0057416749999999999 2024-04-26T01:08:36.769 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.769 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.769 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.769 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063292+0000", 2024-04-26T01:08:36.769 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0913999999999999e-05 2024-04-26T01:08:36.769 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.769 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.769 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.769 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075418+0000", 2024-04-26T01:08:36.769 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012126296999999999 2024-04-26T01:08:36.769 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.769 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.769 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.769 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075521+0000", 2024-04-26T01:08:36.770 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000102885 2024-04-26T01:08:36.770 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.770 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.770 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.770 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.770 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.770 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:08:36.770 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.055795+0000", 2024-04-26T01:08:36.770 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.663557295, 2024-04-26T01:08:36.770 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.02721587, 2024-04-26T01:08:36.770 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.770 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.770 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.770 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.770 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.771 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.771 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055795+0000", 2024-04-26T01:08:36.771 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.771 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.771 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.771 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.771 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055795+0000", 2024-04-26T01:08:36.771 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.771 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.771 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.771 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.771 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055796+0000", 2024-04-26T01:08:36.771 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.057e-06 2024-04-26T01:08:36.771 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.771 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.772 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.772 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.056601+0000", 2024-04-26T01:08:36.772 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00080521399999999995 2024-04-26T01:08:36.772 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.772 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.772 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.772 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063232+0000", 2024-04-26T01:08:36.772 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0066301579999999997 2024-04-26T01:08:36.772 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.772 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.772 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.772 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063246+0000", 2024-04-26T01:08:36.772 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4124000000000001e-05 2024-04-26T01:08:36.772 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.773 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.773 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.773 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.082918+0000", 2024-04-26T01:08:36.773 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.019672486999999999 2024-04-26T01:08:36.773 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.773 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.773 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.773 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.083011+0000", 2024-04-26T01:08:36.773 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2830000000000004e-05 2024-04-26T01:08:36.773 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.773 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.773 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.773 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.773 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.773 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:08:36.774 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.057551+0000", 2024-04-26T01:08:36.774 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.661801044000001, 2024-04-26T01:08:36.774 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.019369432999999998, 2024-04-26T01:08:36.774 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.774 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.774 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.774 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.774 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.774 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.774 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057551+0000", 2024-04-26T01:08:36.774 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.774 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.774 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.774 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.774 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057551+0000", 2024-04-26T01:08:36.775 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.775 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.775 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.775 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.775 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.057553+0000", 2024-04-26T01:08:36.775 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.144e-06 2024-04-26T01:08:36.775 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.775 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.775 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.775 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061928+0000", 2024-04-26T01:08:36.775 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0043754420000000002 2024-04-26T01:08:36.775 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.775 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.775 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.776 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063323+0000", 2024-04-26T01:08:36.776 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001394593 2024-04-26T01:08:36.776 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.776 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.776 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.776 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063333+0000", 2024-04-26T01:08:36.776 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0200000000000001e-05 2024-04-26T01:08:36.776 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.776 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.776 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.776 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076732+0000", 2024-04-26T01:08:36.776 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013399576999999999 2024-04-26T01:08:36.776 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.776 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.776 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.076921+0000", 2024-04-26T01:08:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00018847699999999999 2024-04-26T01:08:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:08:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.061940+0000", 2024-04-26T01:08:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.657412385000001, 2024-04-26T01:08:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.020587278000000001, 2024-04-26T01:08:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.778 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.778 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.778 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061940+0000", 2024-04-26T01:08:36.778 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.778 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.778 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.778 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.778 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061940+0000", 2024-04-26T01:08:36.778 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.778 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.778 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.778 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.778 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.061941+0000", 2024-04-26T01:08:36.778 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.404e-06 2024-04-26T01:08:36.778 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.779 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.779 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.779 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.062953+0000", 2024-04-26T01:08:36.779 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0010116999999999999 2024-04-26T01:08:36.779 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.779 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.779 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.779 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063362+0000", 2024-04-26T01:08:36.779 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000409002 2024-04-26T01:08:36.779 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.779 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.779 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.779 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063372+0000", 2024-04-26T01:08:36.779 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.026e-05 2024-04-26T01:08:36.779 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.780 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.780 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.780 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.082380+0000", 2024-04-26T01:08:36.780 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.019007567999999999 2024-04-26T01:08:36.780 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.780 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.780 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.780 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.082527+0000", 2024-04-26T01:08:36.780 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014734399999999999 2024-04-26T01:08:36.780 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.780 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.780 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.780 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.780 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.781 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s2 58/44 ECSubWrite(tid=309, reqid=osd.4.0:309, at_version=58'58, trim_to=0'0, roll_forward_to=58'54))", 2024-04-26T01:08:36.781 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:24.213069+0000", 2024-04-26T01:08:36.781 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 12.506283112, 2024-04-26T01:08:36.781 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.14087053199999999, 2024-04-26T01:08:36.781 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.781 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.781 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.781 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.781 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.781 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.781 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.213069+0000", 2024-04-26T01:08:36.781 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.781 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.781 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.781 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.782 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.213069+0000", 2024-04-26T01:08:36.782 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.782 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.782 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.782 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.782 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.213070+0000", 2024-04-26T01:08:36.782 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.44e-07 2024-04-26T01:08:36.782 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.782 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.782 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.782 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.213092+0000", 2024-04-26T01:08:36.782 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1849999999999999e-05 2024-04-26T01:08:36.782 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.350821+0000", 2024-04-26T01:08:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.13772947099999999 2024-04-26T01:08:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.350840+0000", 2024-04-26T01:08:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.927e-05 2024-04-26T01:08:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.351016+0000", 2024-04-26T01:08:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017531400000000001 2024-04-26T01:08:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.784 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.784 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.784 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.351047+0000", 2024-04-26T01:08:36.784 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.1099999999999997e-05 2024-04-26T01:08:36.784 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.784 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.784 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.784 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.353853+0000", 2024-04-26T01:08:36.784 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0028058359999999999 2024-04-26T01:08:36.784 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.784 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.784 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.784 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.353940+0000", 2024-04-26T01:08:36.784 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.7146999999999998e-05 2024-04-26T01:08:36.784 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.785 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.785 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.785 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.785 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.785 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs6 58/30 ECSubWrite(tid=318, reqid=client.4521.0:601, at_version=58'41, trim_to=0'0, roll_forward_to=58'39))", 2024-04-26T01:08:36.785 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:24.361342+0000", 2024-04-26T01:08:36.785 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 12.358010767, 2024-04-26T01:08:36.785 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.76301632799999997, 2024-04-26T01:08:36.785 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.785 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.785 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:36.785 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.785 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.785 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.785 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.361342+0000", 2024-04-26T01:08:36.786 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.786 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.786 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.786 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.786 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.361342+0000", 2024-04-26T01:08:36.786 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.786 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.786 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.786 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.786 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.361343+0000", 2024-04-26T01:08:36.786 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.683e-06 2024-04-26T01:08:36.786 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.786 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.786 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.786 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.361642+0000", 2024-04-26T01:08:36.787 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00029905399999999999 2024-04-26T01:08:36.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.787 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.787 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.787 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.121642+0000", 2024-04-26T01:08:36.787 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.75999975099999995 2024-04-26T01:08:36.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.787 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.787 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.787 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.121662+0000", 2024-04-26T01:08:36.787 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9956e-05 2024-04-26T01:08:36.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.787 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.788 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.788 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.121819+0000", 2024-04-26T01:08:36.788 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015640700000000001 2024-04-26T01:08:36.788 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.788 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.788 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.788 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.121838+0000", 2024-04-26T01:08:36.788 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9697000000000001e-05 2024-04-26T01:08:36.788 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.788 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.788 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.788 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.124294+0000", 2024-04-26T01:08:36.788 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0024562329999999999 2024-04-26T01:08:36.788 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.789 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.789 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.789 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.124358+0000", 2024-04-26T01:08:36.789 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.3547000000000007e-05 2024-04-26T01:08:36.789 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.789 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.789 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.789 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.789 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:631 3.cs0 3:31be42eb:::smithi16428812-9:head [read 0~1671168,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-04-26T01:08:36.789 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:25.490167+0000", 2024-04-26T01:08:36.789 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 11.229185692, 2024-04-26T01:08:36.789 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.022116891, 2024-04-26T01:08:36.789 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.790 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.790 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:36.790 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:36.790 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:36.790 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:36.790 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 631 2024-04-26T01:08:36.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.790 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.790 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.790 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.790 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.490167+0000", 2024-04-26T01:08:36.790 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.790 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.791 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.791 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.490167+0000", 2024-04-26T01:08:36.791 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.791 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.791 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.490168+0000", 2024-04-26T01:08:36.791 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1429999999999999e-06 2024-04-26T01:08:36.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.791 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.791 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.490176+0000", 2024-04-26T01:08:36.791 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.5099999999999998e-06 2024-04-26T01:08:36.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.792 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.792 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.792 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.490195+0000", 2024-04-26T01:08:36.792 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8493000000000001e-05 2024-04-26T01:08:36.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.792 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.792 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.792 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.490210+0000", 2024-04-26T01:08:36.792 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5267000000000001e-05 2024-04-26T01:08:36.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.792 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.792 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.792 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.494720+0000", 2024-04-26T01:08:36.792 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0045096959999999997 2024-04-26T01:08:36.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.793 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.793 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:36.793 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.494998+0000", 2024-04-26T01:08:36.793 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00027772700000000002 2024-04-26T01:08:36.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.793 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.793 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.793 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.512284+0000", 2024-04-26T01:08:36.793 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.017286055000000002 2024-04-26T01:08:36.793 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.793 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.793 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.794 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.794 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:632 3.cs0 3:3ca9ce7a:::smithi16428812-23:head [append 0~770048] snapc f=[f,d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-26T01:08:36.794 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:25.499881+0000", 2024-04-26T01:08:36.794 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 11.219471877, 2024-04-26T01:08:36.794 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.022514513, 2024-04-26T01:08:36.794 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.794 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.794 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:36.794 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:36.794 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:36.794 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:36.794 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 632 2024-04-26T01:08:36.794 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.795 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.795 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.795 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.795 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.499881+0000", 2024-04-26T01:08:36.795 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.795 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.795 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.795 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.499881+0000", 2024-04-26T01:08:36.795 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.795 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.795 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.795 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.499884+0000", 2024-04-26T01:08:36.795 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.6940000000000001e-06 2024-04-26T01:08:36.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.796 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.796 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.796 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.501279+0000", 2024-04-26T01:08:36.796 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001394651 2024-04-26T01:08:36.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.796 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.796 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.796 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.501315+0000", 2024-04-26T01:08:36.796 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.5893999999999999e-05 2024-04-26T01:08:36.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.796 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.796 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.796 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.501338+0000", 2024-04-26T01:08:36.796 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3482999999999999e-05 2024-04-26T01:08:36.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.797 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.797 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.797 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.504735+0000", 2024-04-26T01:08:36.797 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.003396568 2024-04-26T01:08:36.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.797 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.797 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:36.797 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.505070+0000", 2024-04-26T01:08:36.797 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000334951 2024-04-26T01:08:36.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.797 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.797 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.797 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.507971+0000", 2024-04-26T01:08:36.797 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002901369 2024-04-26T01:08:36.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.798 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.798 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.798 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.512758+0000", 2024-04-26T01:08:36.798 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.004786434 2024-04-26T01:08:36.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.798 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.798 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:36.798 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.522117+0000", 2024-04-26T01:08:36.798 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0093590789999999993 2024-04-26T01:08:36.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.798 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.798 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.798 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.522395+0000", 2024-04-26T01:08:36.798 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00027839 2024-04-26T01:08:36.799 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.799 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.799 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.799 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:635 3.cs0 3:3ca9ce7a:::smithi16428812-23:head [setxattr _header (57) in=7b] snapc f=[f,d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-26T01:08:36.799 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:25.511495+0000", 2024-04-26T01:08:36.799 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 11.207856978000001, 2024-04-26T01:08:36.799 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.26064569300000001, 2024-04-26T01:08:36.799 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.799 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.799 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:36.799 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:36.799 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:36.800 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:36.800 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 635 2024-04-26T01:08:36.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.800 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.800 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.800 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.800 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511495+0000", 2024-04-26T01:08:36.800 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.800 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.800 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.800 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511495+0000", 2024-04-26T01:08:36.800 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.801 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.801 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.801 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511497+0000", 2024-04-26T01:08:36.801 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2130000000000001e-06 2024-04-26T01:08:36.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.801 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.801 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.801 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511502+0000", 2024-04-26T01:08:36.801 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.4999999999999999e-06 2024-04-26T01:08:36.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.801 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.801 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.801 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511516+0000", 2024-04-26T01:08:36.801 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.402e-05 2024-04-26T01:08:36.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.802 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.802 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511525+0000", 2024-04-26T01:08:36.802 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.0869999999999996e-06 2024-04-26T01:08:36.802 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.802 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.802 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.524239+0000", 2024-04-26T01:08:36.802 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012713296000000001 2024-04-26T01:08:36.802 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.802 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:36.802 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.524539+0000", 2024-04-26T01:08:36.802 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000300501 2024-04-26T01:08:36.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.803 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.803 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:36.803 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.525917+0000", 2024-04-26T01:08:36.803 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0013776839999999999 2024-04-26T01:08:36.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.803 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.803 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:36.803 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.527728+0000", 2024-04-26T01:08:36.803 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0018110229999999999 2024-04-26T01:08:36.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.803 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.803 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:36.803 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.771988+0000", 2024-04-26T01:08:36.803 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.244260692 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.772141+0000", 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015267699999999999 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:636 3.cs0 3:3ca9ce7a:::smithi16428812-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:25.511538+0000", 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 11.207814808, 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.26493635500000001, 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:36.804 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:36.805 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:36.805 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:36.805 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:36.805 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:36.805 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 636 2024-04-26T01:08:36.805 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.805 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:36.805 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.805 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:36.805 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511538+0000", 2024-04-26T01:08:36.805 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.805 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.805 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.805 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:36.805 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511538+0000", 2024-04-26T01:08:36.806 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:36.806 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.806 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.806 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:36.806 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511538+0000", 2024-04-26T01:08:36.806 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8000000000000005e-07 2024-04-26T01:08:36.806 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.806 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.806 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:36.806 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511542+0000", 2024-04-26T01:08:36.806 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.0260000000000001e-06 2024-04-26T01:08:36.806 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.806 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.806 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:36.806 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511556+0000", 2024-04-26T01:08:36.807 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3784000000000001e-05 2024-04-26T01:08:36.807 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.807 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.807 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:36.807 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511576+0000", 2024-04-26T01:08:36.807 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.948e-05 2024-04-26T01:08:36.807 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.807 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.807 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.807 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.526765+0000", 2024-04-26T01:08:36.807 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.015189393000000001 2024-04-26T01:08:36.807 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.807 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.808 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:36.808 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.526913+0000", 2024-04-26T01:08:36.808 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014794299999999999 2024-04-26T01:08:36.808 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.808 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.808 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:36.808 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.772915+0000", 2024-04-26T01:08:36.808 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.24600209100000001 2024-04-26T01:08:36.808 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.808 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.808 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:36.808 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.773181+0000", 2024-04-26T01:08:36.808 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00026546399999999998 2024-04-26T01:08:36.808 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:36.808 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:36.809 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:36.809 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.776474+0000", 2024-04-26T01:08:36.809 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0032933939999999998 2024-04-26T01:08:36.809 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.809 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.809 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.809 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:36.809 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:36.809 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:36.811 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:08:36.812 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:36.807+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:36.911 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:36.911+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:36.940 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:36.940 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:36.940 INFO:teuthology.orchestra.run.smithi012.stdout: "num_ops": 0 2024-04-26T01:08:36.940 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:36.954 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:08:37.012 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:37.011+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:37.054 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.6 to 1 (10000) 2024-04-26T01:08:37.060 INFO:tasks.rados.rados.0.smithi164.stdout:285: done (0 left) 2024-04-26T01:08:37.060 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 33 current snap is 19 2024-04-26T01:08:37.060 INFO:tasks.rados.rados.0.smithi164.stdout:286: seq_num 126 ranges {1916928=573440,2490368=622592,3112960=622592,3735552=311296} 2024-04-26T01:08:37.081 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:37.081 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:37.081 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:08:37.081 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:08:37.081 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:37.083 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-26T01:08:37.095 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:08:37.101 INFO:tasks.rados.rados.0.smithi164.stdout:286: writing smithi16428812-33 from 1916928 to 2490368 tid 1 2024-04-26T01:08:37.111 INFO:tasks.rados.rados.0.smithi164.stdout:286: writing smithi16428812-33 from 2490368 to 3112960 tid 2 2024-04-26T01:08:37.112 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:37.111+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:37.120 INFO:tasks.rados.rados.0.smithi164.stdout:286: writing smithi16428812-33 from 3112960 to 3735552 tid 3 2024-04-26T01:08:37.124 INFO:tasks.rados.rados.0.smithi164.stdout:286: writing smithi16428812-33 from 3735552 to 4046848 tid 4 2024-04-26T01:08:37.124 INFO:tasks.rados.rados.0.smithi164.stdout:287: copy_from oid 19 from oid 10 current snap is 19 2024-04-26T01:08:37.124 INFO:tasks.rados.rados.0.smithi164.stdout:286: finishing write tid 1 to smithi16428812-33 2024-04-26T01:08:37.124 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 5 current snap is 19 2024-04-26T01:08:37.124 INFO:tasks.rados.rados.0.smithi164.stdout:288: seq_num 127 ranges {81920=589824,671744=753664,1425408=475136,1900544=327680} 2024-04-26T01:08:37.131 INFO:tasks.rados.rados.0.smithi164.stdout:288: writing smithi16428812-5 from 81920 to 671744 tid 1 2024-04-26T01:08:37.140 INFO:tasks.rados.rados.0.smithi164.stdout:288: writing smithi16428812-5 from 671744 to 1425408 tid 2 2024-04-26T01:08:37.144 INFO:tasks.rados.rados.0.smithi164.stdout:288: writing smithi16428812-5 from 1425408 to 1900544 tid 3 2024-04-26T01:08:37.147 INFO:tasks.rados.rados.0.smithi164.stdout:288: writing smithi16428812-5 from 1900544 to 2228224 tid 4 2024-04-26T01:08:37.147 INFO:tasks.rados.rados.0.smithi164.stdout:289: rmattr oid 11 current snap is 19 2024-04-26T01:08:37.147 INFO:tasks.rados.rados.0.smithi164.stdout:289: done (3 left) 2024-04-26T01:08:37.147 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 24 current snap is 19 2024-04-26T01:08:37.147 INFO:tasks.rados.rados.0.smithi164.stdout:290: seq_num 128 ranges {2113536=606208,2719744=507904,3227648=573440,3801088=475136} 2024-04-26T01:08:37.167 INFO:tasks.rados.rados.0.smithi164.stdout:290: writing smithi16428812-24 from 2113536 to 2719744 tid 1 2024-04-26T01:08:37.170 INFO:tasks.rados.rados.0.smithi164.stdout:290: writing smithi16428812-24 from 2719744 to 3227648 tid 2 2024-04-26T01:08:37.174 INFO:tasks.rados.rados.0.smithi164.stdout:290: writing smithi16428812-24 from 3227648 to 3801088 tid 3 2024-04-26T01:08:37.177 INFO:tasks.rados.rados.0.smithi164.stdout:290: writing smithi16428812-24 from 3801088 to 4276224 tid 4 2024-04-26T01:08:37.177 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 32 current snap is 19 2024-04-26T01:08:37.177 INFO:tasks.rados.rados.0.smithi164.stdout:286: finishing write tid 2 to smithi16428812-33 2024-04-26T01:08:37.177 INFO:tasks.rados.rados.0.smithi164.stdout:291: seq_num 129 ranges {1982464=622592,2605056=655360,3260416=688128,3948544=212992} 2024-04-26T01:08:37.193 INFO:tasks.rados.rados.0.smithi164.stdout:291: writing smithi16428812-32 from 1982464 to 2605056 tid 1 2024-04-26T01:08:37.196 INFO:tasks.rados.rados.0.smithi164.stdout:291: writing smithi16428812-32 from 2605056 to 3260416 tid 2 2024-04-26T01:08:37.200 INFO:tasks.rados.rados.0.smithi164.stdout:291: writing smithi16428812-32 from 3260416 to 3948544 tid 3 2024-04-26T01:08:37.202 INFO:tasks.rados.rados.0.smithi164.stdout:291: writing smithi16428812-32 from 3948544 to 4161536 tid 4 2024-04-26T01:08:37.202 INFO:tasks.rados.rados.0.smithi164.stdout:292: read oid 27 snap -1 2024-04-26T01:08:37.202 INFO:tasks.rados.rados.0.smithi164.stdout:292: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-26T01:08:37.202 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 36 current snap is 19 2024-04-26T01:08:37.202 INFO:tasks.rados.rados.0.smithi164.stdout:293: seq_num 130 ranges {1490944=638976,2129920=671744,2801664=802816,3604480=81920} 2024-04-26T01:08:37.213 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:37.211+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:37.214 INFO:tasks.rados.rados.0.smithi164.stdout:293: writing smithi16428812-36 from 1490944 to 2129920 tid 1 2024-04-26T01:08:37.218 INFO:tasks.rados.rados.0.smithi164.stdout:293: writing smithi16428812-36 from 2129920 to 2801664 tid 2 2024-04-26T01:08:37.223 INFO:tasks.rados.rados.0.smithi164.stdout:293: writing smithi16428812-36 from 2801664 to 3604480 tid 3 2024-04-26T01:08:37.223 INFO:tasks.rados.rados.0.smithi164.stdout:293: writing smithi16428812-36 from 3604480 to 3686400 tid 4 2024-04-26T01:08:37.223 INFO:tasks.rados.rados.0.smithi164.stdout:294: rollback oid 30 current snap is 19 2024-04-26T01:08:37.224 INFO:tasks.rados.rados.0.smithi164.stdout:294: done (7 left) 2024-04-26T01:08:37.224 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 48 current snap is 19 2024-04-26T01:08:37.224 INFO:tasks.rados.rados.0.smithi164.stdout:295: seq_num 131 ranges {0=655360,655360=425984,1081344=638976,1720320=491520} 2024-04-26T01:08:37.227 INFO:tasks.rados.rados.0.smithi164.stdout:295: writing smithi16428812-48 from 0 to 655360 tid 1 2024-04-26T01:08:37.230 INFO:tasks.rados.rados.0.smithi164.stdout:295: writing smithi16428812-48 from 655360 to 1081344 tid 2 2024-04-26T01:08:37.233 INFO:tasks.rados.rados.0.smithi164.stdout:295: writing smithi16428812-48 from 1081344 to 1720320 tid 3 2024-04-26T01:08:37.236 INFO:tasks.rados.rados.0.smithi164.stdout:295: writing smithi16428812-48 from 1720320 to 2211840 tid 4 2024-04-26T01:08:37.236 INFO:tasks.rados.rados.0.smithi164.stdout:296: rollback oid 39 current snap is 19 2024-04-26T01:08:37.236 INFO:tasks.rados.rados.0.smithi164.stdout:296: done (8 left) 2024-04-26T01:08:37.237 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 35 current snap is 19 2024-04-26T01:08:37.237 INFO:tasks.rados.rados.0.smithi164.stdout:286: finishing write tid 3 to smithi16428812-33 2024-04-26T01:08:37.237 INFO:tasks.rados.rados.0.smithi164.stdout:297: seq_num 132 ranges {3866624=671744,4538368=573440,5111808=753664,5865472=229376} 2024-04-26T01:08:37.238 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:37.238 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:08:37.238 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:08:37.238 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:08:37.238 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.238 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-26T01:08:37.238 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:46.050143+0000", 2024-04-26T01:08:37.238 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 51.187664052999999, 2024-04-26T01:08:37.238 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.36180093800000002, 2024-04-26T01:08:37.238 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.238 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.238 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:37.239 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:37.239 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:08:37.239 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:08:37.239 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 1 2024-04-26T01:08:37.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.239 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.239 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.239 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.239 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050143+0000", 2024-04-26T01:08:37.239 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.239 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.239 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.239 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050143+0000", 2024-04-26T01:08:37.240 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.240 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.240 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.240 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050146+0000", 2024-04-26T01:08:37.240 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7829999999999999e-06 2024-04-26T01:08:37.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.240 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.240 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.240 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050160+0000", 2024-04-26T01:08:37.240 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4314000000000001e-05 2024-04-26T01:08:37.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.240 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.240 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.241 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050182+0000", 2024-04-26T01:08:37.241 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2407000000000001e-05 2024-04-26T01:08:37.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.241 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.241 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.241 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050200+0000", 2024-04-26T01:08:37.241 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7547e-05 2024-04-26T01:08:37.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.241 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.241 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.241 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050332+0000", 2024-04-26T01:08:37.241 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013237700000000001 2024-04-26T01:08:37.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.241 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.241 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for peered", 2024-04-26T01:08:37.242 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050361+0000", 2024-04-26T01:08:37.242 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8704000000000002e-05 2024-04-26T01:08:37.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.242 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.242 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.407786+0000", 2024-04-26T01:08:37.242 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.35742484899999999 2024-04-26T01:08:37.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.242 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:37.242 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.408252+0000", 2024-04-26T01:08:37.242 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00046585099999999998 2024-04-26T01:08:37.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.243 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.243 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:08:37.243 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.408586+0000", 2024-04-26T01:08:37.243 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033444200000000001 2024-04-26T01:08:37.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.243 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.243 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:08:37.243 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.409758+0000", 2024-04-26T01:08:37.243 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001171718 2024-04-26T01:08:37.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.243 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.243 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:08:37.243 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.411769+0000", 2024-04-26T01:08:37.243 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0020110520000000002 2024-04-26T01:08:37.244 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.244 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.244 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:37.244 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.411870+0000", 2024-04-26T01:08:37.244 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000101409 2024-04-26T01:08:37.244 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.244 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.244 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.244 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.411944+0000", 2024-04-26T01:08:37.244 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.3485000000000005e-05 2024-04-26T01:08:37.244 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.244 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.244 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.244 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.245 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.245 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:90 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-26T01:08:37.245 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:46.601265+0000", 2024-04-26T01:08:37.245 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 50.636541831000002, 2024-04-26T01:08:37.245 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98632521799999995, 2024-04-26T01:08:37.245 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.245 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.245 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:37.245 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:37.245 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:08:37.245 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:08:37.245 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 90 2024-04-26T01:08:37.245 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.246 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.246 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.246 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.246 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601265+0000", 2024-04-26T01:08:37.246 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.246 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.246 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.246 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.246 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601265+0000", 2024-04-26T01:08:37.246 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.246 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.246 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.246 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.247 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601267+0000", 2024-04-26T01:08:37.247 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.37e-06 2024-04-26T01:08:37.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.247 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.247 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.247 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601418+0000", 2024-04-26T01:08:37.247 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015154400000000001 2024-04-26T01:08:37.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.247 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.247 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.247 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601436+0000", 2024-04-26T01:08:37.247 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7666999999999999e-05 2024-04-26T01:08:37.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.247 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.247 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.248 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601447+0000", 2024-04-26T01:08:37.248 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1406999999999999e-05 2024-04-26T01:08:37.248 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.248 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.248 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.248 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.602924+0000", 2024-04-26T01:08:37.248 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0014766899999999999 2024-04-26T01:08:37.248 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.248 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.248 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:37.248 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.603071+0000", 2024-04-26T01:08:37.248 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014674600000000001 2024-04-26T01:08:37.248 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.248 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.249 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:08:37.249 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.603406+0000", 2024-04-26T01:08:37.249 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033568500000000001 2024-04-26T01:08:37.249 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.249 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.249 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:08:37.249 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.604963+0000", 2024-04-26T01:08:37.249 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0015563809999999999 2024-04-26T01:08:37.249 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.249 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.249 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:08:37.249 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587422+0000", 2024-04-26T01:08:37.249 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98245948699999996 2024-04-26T01:08:37.249 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.249 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.250 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:37.250 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587520+0000", 2024-04-26T01:08:37.250 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7650000000000002e-05 2024-04-26T01:08:37.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.250 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.250 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.250 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587590+0000", 2024-04-26T01:08:37.250 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.0591000000000002e-05 2024-04-26T01:08:37.250 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.250 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.251 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.251 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.251 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.251 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_repop_reply(client.4188.0:90 1.0 e24/23 ondisk, result = 0)", 2024-04-26T01:08:37.251 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:46.605376+0000", 2024-04-26T01:08:37.251 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 50.632431431000001, 2024-04-26T01:08:37.251 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98225920899999997, 2024-04-26T01:08:37.251 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.251 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.251 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:37.251 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.251 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.251 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.252 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605376+0000", 2024-04-26T01:08:37.252 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.252 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.252 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.252 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605376+0000", 2024-04-26T01:08:37.252 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.252 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.252 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.252 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605377+0000", 2024-04-26T01:08:37.252 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.407e-06 2024-04-26T01:08:37.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.253 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.253 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605384+0000", 2024-04-26T01:08:37.253 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.5299999999999999e-06 2024-04-26T01:08:37.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.253 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.253 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587134+0000", 2024-04-26T01:08:37.253 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98174979699999998 2024-04-26T01:08:37.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.255 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.255 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587156+0000", 2024-04-26T01:08:37.255 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1704000000000001e-05 2024-04-26T01:08:37.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.255 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.255 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587391+0000", 2024-04-26T01:08:37.255 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000235052 2024-04-26T01:08:37.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.256 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.256 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:37.256 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587412+0000", 2024-04-26T01:08:37.256 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0630999999999999e-05 2024-04-26T01:08:37.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.256 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.256 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.256 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587635+0000", 2024-04-26T01:08:37.256 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000223088 2024-04-26T01:08:37.256 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.256 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.256 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.257 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:145 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-26T01:08:37.257 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:47.981007+0000", 2024-04-26T01:08:37.257 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 49.256799569000002, 2024-04-26T01:08:37.257 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.057997620999999999, 2024-04-26T01:08:37.257 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.257 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.257 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:37.257 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:37.257 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:08:37.257 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:08:37.257 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 145 2024-04-26T01:08:37.257 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.257 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.258 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.258 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.258 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981007+0000", 2024-04-26T01:08:37.258 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.258 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.258 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.258 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981007+0000", 2024-04-26T01:08:37.258 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.259 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.259 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.259 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981009+0000", 2024-04-26T01:08:37.259 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.204e-06 2024-04-26T01:08:37.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.259 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.259 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.259 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981146+0000", 2024-04-26T01:08:37.259 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013786000000000001 2024-04-26T01:08:37.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.259 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.259 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.259 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981165+0000", 2024-04-26T01:08:37.259 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8542999999999998e-05 2024-04-26T01:08:37.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.260 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.260 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981177+0000", 2024-04-26T01:08:37.260 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1787000000000001e-05 2024-04-26T01:08:37.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.260 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.260 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.982190+0000", 2024-04-26T01:08:37.260 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0010130180000000001 2024-04-26T01:08:37.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.260 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:37.260 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.982236+0000", 2024-04-26T01:08:37.260 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.6445e-05 2024-04-26T01:08:37.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.261 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:08:37.261 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.982311+0000", 2024-04-26T01:08:37.261 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.4657e-05 2024-04-26T01:08:37.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.261 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:08:37.261 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.983568+0000", 2024-04-26T01:08:37.261 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001257172 2024-04-26T01:08:37.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.261 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:08:37.261 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.038854+0000", 2024-04-26T01:08:37.262 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.055285918000000003 2024-04-26T01:08:37.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.262 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.262 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:37.262 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.038941+0000", 2024-04-26T01:08:37.262 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.6971999999999993e-05 2024-04-26T01:08:37.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.262 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.262 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.262 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.039005+0000", 2024-04-26T01:08:37.262 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.4045000000000001e-05 2024-04-26T01:08:37.262 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.262 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.262 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.263 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.263 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:15 3.2s0 3:49548768:::smithi16428812-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:37.263 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.304518+0000", 2024-04-26T01:08:37.263 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.933288693999998, 2024-04-26T01:08:37.263 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.037662963000000001, 2024-04-26T01:08:37.263 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.263 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.263 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:37.263 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:37.263 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:37.263 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:37.263 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 15 2024-04-26T01:08:37.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.263 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.264 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.264 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.264 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304518+0000", 2024-04-26T01:08:37.264 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.264 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.264 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.264 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304518+0000", 2024-04-26T01:08:37.264 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.264 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.264 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.265 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304520+0000", 2024-04-26T01:08:37.265 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.296e-06 2024-04-26T01:08:37.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.265 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.265 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.265 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304527+0000", 2024-04-26T01:08:37.265 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.25e-06 2024-04-26T01:08:37.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.265 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.265 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.265 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304547+0000", 2024-04-26T01:08:37.265 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0307999999999999e-05 2024-04-26T01:08:37.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.265 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.266 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.266 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304564+0000", 2024-04-26T01:08:37.266 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6529999999999999e-05 2024-04-26T01:08:37.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.266 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.266 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.266 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313033+0000", 2024-04-26T01:08:37.266 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0084690919999999992 2024-04-26T01:08:37.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.266 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.266 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:37.266 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313289+0000", 2024-04-26T01:08:37.266 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025617400000000002 2024-04-26T01:08:37.267 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.267 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.267 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.267 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.334874+0000", 2024-04-26T01:08:37.267 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.021585066999999999 2024-04-26T01:08:37.267 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.267 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.267 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:37.267 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.335136+0000", 2024-04-26T01:08:37.267 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00026158000000000003 2024-04-26T01:08:37.267 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.267 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.267 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.267 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.342181+0000", 2024-04-26T01:08:37.268 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0070456659999999999 2024-04-26T01:08:37.268 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.268 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.268 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.268 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.268 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:33 3.2s0 3:410d186f:::smithi16428812-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:37.268 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.328026+0000", 2024-04-26T01:08:37.268 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.909781221999999, 2024-04-26T01:08:37.268 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.065568274999999995, 2024-04-26T01:08:37.268 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.268 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.268 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:37.268 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 33 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328026+0000", 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328026+0000", 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.269 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.270 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.270 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328028+0000", 2024-04-26T01:08:37.270 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.357e-06 2024-04-26T01:08:37.270 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.270 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.270 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.270 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328040+0000", 2024-04-26T01:08:37.270 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2257e-05 2024-04-26T01:08:37.270 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.270 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.270 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.270 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328089+0000", 2024-04-26T01:08:37.270 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.8164000000000002e-05 2024-04-26T01:08:37.270 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.271 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.271 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328112+0000", 2024-04-26T01:08:37.271 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3444000000000001e-05 2024-04-26T01:08:37.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.271 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.271 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328852+0000", 2024-04-26T01:08:37.271 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00074019899999999998 2024-04-26T01:08:37.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.271 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:37.271 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328981+0000", 2024-04-26T01:08:37.272 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012923500000000001 2024-04-26T01:08:37.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.272 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.272 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.272 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.339980+0000", 2024-04-26T01:08:37.272 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010998232 2024-04-26T01:08:37.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.272 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.272 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:37.272 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.340144+0000", 2024-04-26T01:08:37.272 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016476699999999999 2024-04-26T01:08:37.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.272 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.273 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.273 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.393594+0000", 2024-04-26T01:08:37.273 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.053449620000000003 2024-04-26T01:08:37.273 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.273 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.273 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.273 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.273 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 33/30 ECSubWriteReply(tid=162, last_complete=33'4, committed=1, applied=1))", 2024-04-26T01:08:37.273 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.341290+0000", 2024-04-26T01:08:37.273 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.896517453000001, 2024-04-26T01:08:37.273 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.050669963999999998, 2024-04-26T01:08:37.273 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.274 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.274 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:37.274 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.274 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.274 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.274 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.341290+0000", 2024-04-26T01:08:37.274 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.274 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.274 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.274 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.341290+0000", 2024-04-26T01:08:37.274 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.274 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.275 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.275 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.341291+0000", 2024-04-26T01:08:37.275 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.237e-06 2024-04-26T01:08:37.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.275 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.275 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.275 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.341301+0000", 2024-04-26T01:08:37.275 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0266e-05 2024-04-26T01:08:37.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.275 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.275 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.275 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.388586+0000", 2024-04-26T01:08:37.275 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.047284633 2024-04-26T01:08:37.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.276 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.276 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.276 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.388599+0000", 2024-04-26T01:08:37.276 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3335e-05 2024-04-26T01:08:37.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.276 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.276 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.276 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.391887+0000", 2024-04-26T01:08:37.276 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0032881400000000002 2024-04-26T01:08:37.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.276 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.276 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.276 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.391959+0000", 2024-04-26T01:08:37.277 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.2353000000000003e-05 2024-04-26T01:08:37.277 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.277 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.277 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.277 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.277 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.277 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:79 3.2s0 3:4b558697:::smithi16428812-27:head [append 0~442368,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:37.277 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.384757+0000", 2024-04-26T01:08:37.277 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.853050359999997, 2024-04-26T01:08:37.277 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.050291098999999999, 2024-04-26T01:08:37.277 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.277 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.277 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:37.277 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:37.278 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:37.278 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:37.278 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 79 2024-04-26T01:08:37.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.278 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.278 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.278 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.278 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.384757+0000", 2024-04-26T01:08:37.278 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.278 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.278 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.278 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.384757+0000", 2024-04-26T01:08:37.278 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.279 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.279 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.279 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.384759+0000", 2024-04-26T01:08:37.279 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.553e-06 2024-04-26T01:08:37.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.279 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.279 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.279 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385233+0000", 2024-04-26T01:08:37.279 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00047340000000000001 2024-04-26T01:08:37.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.279 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.279 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.279 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385261+0000", 2024-04-26T01:08:37.280 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8405999999999999e-05 2024-04-26T01:08:37.280 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.280 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.280 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.280 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385272+0000", 2024-04-26T01:08:37.280 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0951000000000001e-05 2024-04-26T01:08:37.280 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.280 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.280 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.280 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385543+0000", 2024-04-26T01:08:37.280 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00027136800000000002 2024-04-26T01:08:37.281 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.281 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.281 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:37.281 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385733+0000", 2024-04-26T01:08:37.281 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00018952800000000001 2024-04-26T01:08:37.281 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.281 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.281 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:37.281 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.387474+0000", 2024-04-26T01:08:37.281 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001741156 2024-04-26T01:08:37.281 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.281 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.281 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:37.281 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.392886+0000", 2024-04-26T01:08:37.282 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0054121239999999999 2024-04-26T01:08:37.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.282 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.282 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:37.282 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.434904+0000", 2024-04-26T01:08:37.282 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.042017911999999998 2024-04-26T01:08:37.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.282 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.282 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.282 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.435048+0000", 2024-04-26T01:08:37.282 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014370100000000001 2024-04-26T01:08:37.282 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.282 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.282 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.283 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.283 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:80 3.2s0 3:4b558697:::smithi16428812-27:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:37.283 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.385307+0000", 2024-04-26T01:08:37.283 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.852499895999998, 2024-04-26T01:08:37.283 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.050590553000000003, 2024-04-26T01:08:37.283 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.283 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.283 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:37.283 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:37.283 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:37.283 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:37.283 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 80 2024-04-26T01:08:37.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.284 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.284 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.284 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.284 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385307+0000", 2024-04-26T01:08:37.284 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.464 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.464 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.464 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385307+0000", 2024-04-26T01:08:37.464 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.464 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.464 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.464 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.464 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385308+0000", 2024-04-26T01:08:37.464 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2099999999999995e-07 2024-04-26T01:08:37.464 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.464 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.464 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.465 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385315+0000", 2024-04-26T01:08:37.465 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.4580000000000004e-06 2024-04-26T01:08:37.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.465 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.465 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385330+0000", 2024-04-26T01:08:37.465 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4188e-05 2024-04-26T01:08:37.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.465 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.465 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385338+0000", 2024-04-26T01:08:37.465 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.2190000000000004e-06 2024-04-26T01:08:37.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.466 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.466 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.466 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.390117+0000", 2024-04-26T01:08:37.466 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0047786770000000003 2024-04-26T01:08:37.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.466 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.466 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:37.466 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.390334+0000", 2024-04-26T01:08:37.466 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00021722000000000001 2024-04-26T01:08:37.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.466 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.466 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:37.466 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.391426+0000", 2024-04-26T01:08:37.466 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0010922429999999999 2024-04-26T01:08:37.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.467 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:37.467 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.393633+0000", 2024-04-26T01:08:37.467 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0022069139999999999 2024-04-26T01:08:37.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.467 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:37.467 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.435677+0000", 2024-04-26T01:08:37.467 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.042043589999999999 2024-04-26T01:08:37.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.467 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.468 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.435898+0000", 2024-04-26T01:08:37.468 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022112299999999999 2024-04-26T01:08:37.468 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.468 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.468 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.468 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:81 3.2s0 3:4b558697:::smithi16428812-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:37.468 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.385353+0000", 2024-04-26T01:08:37.468 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.852453934000003, 2024-04-26T01:08:37.468 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.12263547800000001, 2024-04-26T01:08:37.468 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.469 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.469 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:37.469 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:37.469 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:37.469 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:37.469 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 81 2024-04-26T01:08:37.469 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.469 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.469 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.469 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.469 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385353+0000", 2024-04-26T01:08:37.469 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.469 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.469 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.469 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.470 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385353+0000", 2024-04-26T01:08:37.470 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.470 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.470 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.470 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.470 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385354+0000", 2024-04-26T01:08:37.470 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.6300000000000004e-07 2024-04-26T01:08:37.470 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.470 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.471 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.471 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385357+0000", 2024-04-26T01:08:37.471 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.4989999999999999e-06 2024-04-26T01:08:37.471 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.471 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.471 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.471 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385381+0000", 2024-04-26T01:08:37.471 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3643999999999999e-05 2024-04-26T01:08:37.471 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.471 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.471 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.471 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385390+0000", 2024-04-26T01:08:37.471 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.9090000000000006e-06 2024-04-26T01:08:37.471 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.472 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.472 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.472 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.392442+0000", 2024-04-26T01:08:37.472 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.007052455 2024-04-26T01:08:37.472 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.472 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.472 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:37.472 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.392642+0000", 2024-04-26T01:08:37.472 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000199662 2024-04-26T01:08:37.472 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.473 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.473 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.473 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.436978+0000", 2024-04-26T01:08:37.473 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.044336042999999999 2024-04-26T01:08:37.473 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.473 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.473 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:37.473 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.437074+0000", 2024-04-26T01:08:37.473 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5700999999999997e-05 2024-04-26T01:08:37.473 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.473 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.473 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.473 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.507989+0000", 2024-04-26T01:08:37.473 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.070914701999999996 2024-04-26T01:08:37.473 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.474 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.474 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.474 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.474 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.474 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 33/30 ECSubWriteReply(tid=164, last_complete=33'5, committed=1, applied=1))", 2024-04-26T01:08:37.474 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.393020+0000", 2024-04-26T01:08:37.474 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.844786767000002, 2024-04-26T01:08:37.474 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.042042138, 2024-04-26T01:08:37.474 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.474 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.474 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:37.474 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.474 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.474 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.475 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.393020+0000", 2024-04-26T01:08:37.475 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.475 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.475 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.475 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.475 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.393020+0000", 2024-04-26T01:08:37.475 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.475 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.475 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.475 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.475 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.393022+0000", 2024-04-26T01:08:37.475 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5099999999999999e-06 2024-04-26T01:08:37.475 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.475 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.475 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.476 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.393033+0000", 2024-04-26T01:08:37.476 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1430000000000001e-05 2024-04-26T01:08:37.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.476 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.476 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.476 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.434612+0000", 2024-04-26T01:08:37.476 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.041578801999999998 2024-04-26T01:08:37.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.476 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.476 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.476 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.434628+0000", 2024-04-26T01:08:37.476 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6524999999999998e-05 2024-04-26T01:08:37.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.477 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.477 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.477 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.434756+0000", 2024-04-26T01:08:37.477 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012755199999999999 2024-04-26T01:08:37.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.477 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.477 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.477 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.435062+0000", 2024-04-26T01:08:37.477 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00030631899999999999 2024-04-26T01:08:37.477 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.477 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.477 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.477 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.477 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.2s0 33/30 ECSubReadReply(tid=169, attrs_read=0))", 2024-04-26T01:08:37.478 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.443824+0000", 2024-04-26T01:08:37.478 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.793982874000001, 2024-04-26T01:08:37.478 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.064178733000000002, 2024-04-26T01:08:37.478 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.478 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.478 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:37.478 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.478 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.478 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.478 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.443824+0000", 2024-04-26T01:08:37.478 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.478 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.478 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.478 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.479 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.443824+0000", 2024-04-26T01:08:37.479 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.479 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.479 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.443825+0000", 2024-04-26T01:08:37.479 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.266e-06 2024-04-26T01:08:37.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.479 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.479 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.443845+0000", 2024-04-26T01:08:37.479 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9236e-05 2024-04-26T01:08:37.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.479 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.480 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.505240+0000", 2024-04-26T01:08:37.480 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.061395523 2024-04-26T01:08:37.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.480 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.480 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.480 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.505256+0000", 2024-04-26T01:08:37.480 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6101999999999999e-05 2024-04-26T01:08:37.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.480 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.480 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.480 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.507809+0000", 2024-04-26T01:08:37.480 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0025529820000000001 2024-04-26T01:08:37.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.480 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.481 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.481 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508003+0000", 2024-04-26T01:08:37.481 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019362399999999999 2024-04-26T01:08:37.481 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.481 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.481 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.481 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.481 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:166 3.2s0 3:410d186f:::smithi16428812-11:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:37.481 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.504084+0000", 2024-04-26T01:08:37.481 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.733722769000003, 2024-04-26T01:08:37.481 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.062718093000000003, 2024-04-26T01:08:37.481 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.481 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.481 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:37.482 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:37.482 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:37.482 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:37.482 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 166 2024-04-26T01:08:37.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.482 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.482 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.482 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504084+0000", 2024-04-26T01:08:37.482 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.482 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.482 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504084+0000", 2024-04-26T01:08:37.483 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.483 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.483 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504086+0000", 2024-04-26T01:08:37.483 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8819999999999999e-06 2024-04-26T01:08:37.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.483 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.483 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504689+0000", 2024-04-26T01:08:37.483 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000602634 2024-04-26T01:08:37.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.483 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.483 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504716+0000", 2024-04-26T01:08:37.484 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6995000000000002e-05 2024-04-26T01:08:37.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.484 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.484 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504732+0000", 2024-04-26T01:08:37.484 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5896000000000002e-05 2024-04-26T01:08:37.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.484 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.484 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504855+0000", 2024-04-26T01:08:37.484 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012368000000000001 2024-04-26T01:08:37.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.484 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:37.484 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.505004+0000", 2024-04-26T01:08:37.485 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000148515 2024-04-26T01:08:37.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.485 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:37.485 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.506133+0000", 2024-04-26T01:08:37.485 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001129625 2024-04-26T01:08:37.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.485 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:37.485 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508806+0000", 2024-04-26T01:08:37.485 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0026721760000000001 2024-04-26T01:08:37.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.485 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:37.485 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566633+0000", 2024-04-26T01:08:37.486 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.057827207999999998 2024-04-26T01:08:37.486 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.486 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.486 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566802+0000", 2024-04-26T01:08:37.486 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016948199999999999 2024-04-26T01:08:37.486 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.486 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.486 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.486 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.486 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:167 3.2s0 3:410d186f:::smithi16428812-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:37.486 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.504769+0000", 2024-04-26T01:08:37.486 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.733038325999999, 2024-04-26T01:08:37.487 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.18953119900000001, 2024-04-26T01:08:37.487 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.487 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.487 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:37.487 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:37.487 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:37.487 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:37.487 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 167 2024-04-26T01:08:37.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.487 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.487 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.487 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.487 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504769+0000", 2024-04-26T01:08:37.487 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.488 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.488 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.488 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504769+0000", 2024-04-26T01:08:37.488 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.488 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.488 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.488 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504770+0000", 2024-04-26T01:08:37.488 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.356e-06 2024-04-26T01:08:37.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.488 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.488 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.488 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504777+0000", 2024-04-26T01:08:37.488 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.8140000000000003e-06 2024-04-26T01:08:37.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.489 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.489 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.489 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504791+0000", 2024-04-26T01:08:37.489 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4267000000000001e-05 2024-04-26T01:08:37.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.489 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.489 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.489 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504807+0000", 2024-04-26T01:08:37.489 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6030000000000001e-05 2024-04-26T01:08:37.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.489 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.489 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.490 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.506957+0000", 2024-04-26T01:08:37.490 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0021496549999999999 2024-04-26T01:08:37.490 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.490 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.490 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:37.490 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.507039+0000", 2024-04-26T01:08:37.490 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1959000000000006e-05 2024-04-26T01:08:37.490 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.490 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.490 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:37.490 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.507462+0000", 2024-04-26T01:08:37.490 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00042285800000000002 2024-04-26T01:08:37.490 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.490 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.491 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:37.491 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.511679+0000", 2024-04-26T01:08:37.491 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0042174209999999998 2024-04-26T01:08:37.491 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.491 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.491 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:37.491 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694149+0000", 2024-04-26T01:08:37.491 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.18246974999999999 2024-04-26T01:08:37.491 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.491 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.491 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.491 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694300+0000", 2024-04-26T01:08:37.491 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000151089 2024-04-26T01:08:37.491 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.492 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.492 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.492 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.492 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.492 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:168 3.2s0 3:410d186f:::smithi16428812-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:37.492 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.504832+0000", 2024-04-26T01:08:37.492 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.732974874999996, 2024-04-26T01:08:37.492 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.19389620699999999, 2024-04-26T01:08:37.492 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.492 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.492 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:37.492 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:37.492 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:37.492 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:37.493 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 168 2024-04-26T01:08:37.493 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.493 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.493 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.493 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.493 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504832+0000", 2024-04-26T01:08:37.493 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.493 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.493 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.493 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.493 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504832+0000", 2024-04-26T01:08:37.493 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.493 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.494 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.494 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.494 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504833+0000", 2024-04-26T01:08:37.494 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5199999999999995e-07 2024-04-26T01:08:37.494 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.494 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.494 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.494 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504838+0000", 2024-04-26T01:08:37.494 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.1560000000000003e-06 2024-04-26T01:08:37.494 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.494 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.494 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.494 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504853+0000", 2024-04-26T01:08:37.494 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.469e-05 2024-04-26T01:08:37.494 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.495 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.495 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.495 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504865+0000", 2024-04-26T01:08:37.495 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1707e-05 2024-04-26T01:08:37.495 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.495 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.495 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.495 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508279+0000", 2024-04-26T01:08:37.495 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0034144650000000002 2024-04-26T01:08:37.495 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.495 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.495 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:37.495 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508350+0000", 2024-04-26T01:08:37.495 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1166000000000003e-05 2024-04-26T01:08:37.496 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.496 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.496 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.496 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694397+0000", 2024-04-26T01:08:37.496 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.18604716299999999 2024-04-26T01:08:37.496 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.496 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.496 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:37.496 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694544+0000", 2024-04-26T01:08:37.496 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014672200000000001 2024-04-26T01:08:37.496 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.496 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.496 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.496 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.698728+0000", 2024-04-26T01:08:37.497 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0041841860000000003 2024-04-26T01:08:37.497 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.497 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.497 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.497 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.497 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.497 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 33/30 ECSubWriteReply(tid=173, last_complete=33'8, committed=1, applied=1))", 2024-04-26T01:08:37.497 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.508639+0000", 2024-04-26T01:08:37.497 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.729167996000001, 2024-04-26T01:08:37.497 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.058189232, 2024-04-26T01:08:37.497 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.497 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.497 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:37.497 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.497 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.498 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.498 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508639+0000", 2024-04-26T01:08:37.498 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.498 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.498 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.498 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.498 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508639+0000", 2024-04-26T01:08:37.498 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.498 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.498 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.498 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.498 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508640+0000", 2024-04-26T01:08:37.498 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.9999999999999997e-07 2024-04-26T01:08:37.499 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.499 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.499 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.499 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508646+0000", 2024-04-26T01:08:37.499 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.0580000000000003e-06 2024-04-26T01:08:37.499 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.499 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.499 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.499 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566182+0000", 2024-04-26T01:08:37.499 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.057536449000000003 2024-04-26T01:08:37.499 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.499 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.499 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.499 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566201+0000", 2024-04-26T01:08:37.499 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8904000000000001e-05 2024-04-26T01:08:37.500 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.500 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.500 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.500 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566315+0000", 2024-04-26T01:08:37.500 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00011435800000000001 2024-04-26T01:08:37.500 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.500 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.500 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.500 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566828+0000", 2024-04-26T01:08:37.500 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00051276299999999996 2024-04-26T01:08:37.500 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.500 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.500 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.500 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.500 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.501 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:191 3.2s0 3:4b558697:::smithi16428812-27:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:37.501 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.563295+0000", 2024-04-26T01:08:37.501 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.674512061000001, 2024-04-26T01:08:37.501 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.13397727300000001, 2024-04-26T01:08:37.501 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.501 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.501 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:37.501 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:37.501 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:37.501 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:37.501 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 191 2024-04-26T01:08:37.501 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.501 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.501 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.502 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.502 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563295+0000", 2024-04-26T01:08:37.502 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.502 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.502 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.502 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.502 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563295+0000", 2024-04-26T01:08:37.502 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.502 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.502 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.502 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.502 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563297+0000", 2024-04-26T01:08:37.502 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4140000000000001e-06 2024-04-26T01:08:37.502 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.503 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.503 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.503 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563318+0000", 2024-04-26T01:08:37.503 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0211e-05 2024-04-26T01:08:37.503 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.503 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.503 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.503 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563342+0000", 2024-04-26T01:08:37.503 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4266000000000001e-05 2024-04-26T01:08:37.503 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.503 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.503 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.503 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563353+0000", 2024-04-26T01:08:37.503 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0886e-05 2024-04-26T01:08:37.504 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.504 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.504 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.504 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563479+0000", 2024-04-26T01:08:37.504 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000126651 2024-04-26T01:08:37.504 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.504 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.504 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:37.504 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563655+0000", 2024-04-26T01:08:37.504 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000175713 2024-04-26T01:08:37.504 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.504 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.504 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:37.504 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.564447+0000", 2024-04-26T01:08:37.505 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00079165500000000005 2024-04-26T01:08:37.505 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.505 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.505 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:37.505 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.565861+0000", 2024-04-26T01:08:37.505 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001414595 2024-04-26T01:08:37.505 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.505 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.505 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:37.505 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.697131+0000", 2024-04-26T01:08:37.505 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.13126945700000001 2024-04-26T01:08:37.505 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.505 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.505 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.506 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.697272+0000", 2024-04-26T01:08:37.506 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014142500000000001 2024-04-26T01:08:37.506 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.506 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.506 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.506 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.506 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.506 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 33/30 ECSubWriteReply(tid=174, last_complete=33'9, committed=1, applied=1))", 2024-04-26T01:08:37.506 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.566047+0000", 2024-04-26T01:08:37.506 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 37.671759969999997, 2024-04-26T01:08:37.506 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.128277063, 2024-04-26T01:08:37.506 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.506 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.506 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:37.507 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.507 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.507 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.507 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566047+0000", 2024-04-26T01:08:37.507 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.507 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.507 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.507 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.507 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566047+0000", 2024-04-26T01:08:37.507 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.507 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.507 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.507 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.507 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566048+0000", 2024-04-26T01:08:37.508 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.316e-06 2024-04-26T01:08:37.508 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.508 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.508 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.508 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566138+0000", 2024-04-26T01:08:37.508 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.9281000000000001e-05 2024-04-26T01:08:37.508 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.508 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.508 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.508 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.693854+0000", 2024-04-26T01:08:37.508 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.12771605 2024-04-26T01:08:37.508 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.508 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.508 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.509 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.693868+0000", 2024-04-26T01:08:37.509 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3910000000000001e-05 2024-04-26T01:08:37.509 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.509 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.509 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.509 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694017+0000", 2024-04-26T01:08:37.509 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000149865 2024-04-26T01:08:37.509 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.509 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.509 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.509 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694324+0000", 2024-04-26T01:08:37.509 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00030664100000000002 2024-04-26T01:08:37.509 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.509 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.509 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.510 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.510 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.510 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 38/30 ECSubWriteReply(tid=190, last_complete=38'7, committed=1, applied=1))", 2024-04-26T01:08:37.510 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.497964+0000", 2024-04-26T01:08:37.510 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 32.73984265, 2024-04-26T01:08:37.510 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.81206265399999999, 2024-04-26T01:08:37.510 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.510 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.510 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:37.510 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.510 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.510 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.510 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.497964+0000", 2024-04-26T01:08:37.510 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.511 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.511 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.511 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.497964+0000", 2024-04-26T01:08:37.511 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.511 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.511 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.511 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.497966+0000", 2024-04-26T01:08:37.511 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.274e-06 2024-04-26T01:08:37.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.511 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.511 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.511 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.497979+0000", 2024-04-26T01:08:37.512 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3127e-05 2024-04-26T01:08:37.512 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.512 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.512 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.512 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.309733+0000", 2024-04-26T01:08:37.512 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.81175416700000003 2024-04-26T01:08:37.512 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.512 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.512 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.512 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.309746+0000", 2024-04-26T01:08:37.512 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3523000000000001e-05 2024-04-26T01:08:37.512 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.512 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.512 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.512 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.309899+0000", 2024-04-26T01:08:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015288199999999999 2024-04-26T01:08:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.310027+0000", 2024-04-26T01:08:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012768100000000001 2024-04-26T01:08:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.fs3 50/44 ECSubWrite(tid=225, reqid=client.4521.0:494, at_version=50'41, trim_to=0'0, roll_forward_to=50'38))", 2024-04-26T01:08:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.091583+0000", 2024-04-26T01:08:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.146224024999999, 2024-04-26T01:08:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.14404377500000001, 2024-04-26T01:08:37.514 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:37.514 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:37.514 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:37.514 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:37.514 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.514 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:37.514 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.091583+0000", 2024-04-26T01:08:37.514 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.514 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.514 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.514 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:37.514 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.091583+0000", 2024-04-26T01:08:37.514 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:37.514 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.515 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.515 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:37.515 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.091585+0000", 2024-04-26T01:08:37.515 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.075e-06 2024-04-26T01:08:37.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.515 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.515 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:37.515 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.091712+0000", 2024-04-26T01:08:37.515 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012731099999999999 2024-04-26T01:08:37.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.515 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.515 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:37.515 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.233513+0000", 2024-04-26T01:08:37.515 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.14180088399999999 2024-04-26T01:08:37.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.233530+0000", 2024-04-26T01:08:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6390000000000001e-05 2024-04-26T01:08:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.233655+0000", 2024-04-26T01:08:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000125341 2024-04-26T01:08:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.233671+0000", 2024-04-26T01:08:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6515999999999999e-05 2024-04-26T01:08:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.517 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.517 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:37.517 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.235542+0000", 2024-04-26T01:08:37.517 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001870125 2024-04-26T01:08:37.517 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:37.517 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:37.517 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:37.517 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.235627+0000", 2024-04-26T01:08:37.517 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.5133e-05 2024-04-26T01:08:37.517 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.517 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.517 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.517 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:37.517 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:37.517 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:37.520 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:08:37.522 INFO:tasks.rados.rados.0.smithi164.stdout:297: writing smithi16428812-35 from 3866624 to 4538368 tid 1 2024-04-26T01:08:37.522 INFO:tasks.rados.rados.0.smithi164.stdout:297: writing smithi16428812-35 from 4538368 to 5111808 tid 2 2024-04-26T01:08:37.522 INFO:tasks.rados.rados.0.smithi164.stdout:297: writing smithi16428812-35 from 5111808 to 5865472 tid 3 2024-04-26T01:08:37.522 INFO:tasks.rados.rados.0.smithi164.stdout:297: writing smithi16428812-35 from 5865472 to 6094848 tid 4 2024-04-26T01:08:37.522 INFO:tasks.rados.rados.0.smithi164.stdout:298: snap_create 2024-04-26T01:08:37.522 INFO:tasks.rados.rados.0.smithi164.stdout:286: finishing write tid 4 to smithi16428812-33 2024-04-26T01:08:37.522 INFO:tasks.rados.rados.0.smithi164.stdout:286: finishing write tid 5 to smithi16428812-33 2024-04-26T01:08:37.522 INFO:tasks.rados.rados.0.smithi164.stdout:288: finishing write tid 1 to smithi16428812-5 2024-04-26T01:08:37.523 INFO:tasks.rados.rados.0.smithi164.stdout:287: finishing copy_from racing read to smithi16428812-19 2024-04-26T01:08:37.523 INFO:tasks.rados.rados.0.smithi164.stdout:288: finishing write tid 2 to smithi16428812-5 2024-04-26T01:08:37.523 INFO:tasks.rados.rados.0.smithi164.stdout:287: finishing copy_from to smithi16428812-19 2024-04-26T01:08:37.523 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 19 v 28 (ObjNum 77 snap 5 seq_num 77) dirty exists 2024-04-26T01:08:37.523 INFO:tasks.rados.rados.0.smithi164.stdout:288: finishing write tid 3 to smithi16428812-5 2024-04-26T01:08:37.523 INFO:tasks.rados.rados.0.smithi164.stdout:288: finishing write tid 4 to smithi16428812-5 2024-04-26T01:08:37.523 INFO:tasks.rados.rados.0.smithi164.stdout:286: finishing write tid 6 to smithi16428812-33 2024-04-26T01:08:37.523 INFO:tasks.rados.rados.0.smithi164.stdout:286: oid 33 updating version 0 to 83 2024-04-26T01:08:37.523 INFO:tasks.rados.rados.0.smithi164.stdout:286: oid 33 version 83 is already newer than 79 2024-04-26T01:08:37.523 INFO:tasks.rados.rados.0.smithi164.stdout:286: oid 33 updating version 83 to 84 2024-04-26T01:08:37.523 INFO:tasks.rados.rados.0.smithi164.stdout:286: oid 33 version 84 is already newer than 82 2024-04-26T01:08:37.523 INFO:tasks.rados.rados.0.smithi164.stdout:286: oid 33 updating version 84 to 85 2024-04-26T01:08:37.523 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 33 v 85 (ObjNum 126 snap 19 seq_num 126) dirty exists 2024-04-26T01:08:37.523 INFO:tasks.rados.rados.0.smithi164.stdout:286: left oid 33 (ObjNum 126 snap 19 seq_num 126) 2024-04-26T01:08:37.524 INFO:tasks.rados.rados.0.smithi164.stdout:290: finishing write tid 1 to smithi16428812-24 2024-04-26T01:08:37.524 INFO:tasks.rados.rados.0.smithi164.stdout:290: finishing write tid 2 to smithi16428812-24 2024-04-26T01:08:37.524 INFO:tasks.rados.rados.0.smithi164.stdout:290: finishing write tid 3 to smithi16428812-24 2024-04-26T01:08:37.524 INFO:tasks.rados.rados.0.smithi164.stdout:290: finishing write tid 4 to smithi16428812-24 2024-04-26T01:08:37.524 INFO:tasks.rados.rados.0.smithi164.stdout:290: finishing write tid 5 to smithi16428812-24 2024-04-26T01:08:37.524 INFO:tasks.rados.rados.0.smithi164.stdout:290: finishing write tid 6 to smithi16428812-24 2024-04-26T01:08:37.524 INFO:tasks.rados.rados.0.smithi164.stdout:290: oid 24 updating version 0 to 58 2024-04-26T01:08:37.524 INFO:tasks.rados.rados.0.smithi164.stdout:290: oid 24 version 58 is already newer than 57 2024-04-26T01:08:37.524 INFO:tasks.rados.rados.0.smithi164.stdout:290: oid 24 version 58 is already newer than 55 2024-04-26T01:08:37.524 INFO:tasks.rados.rados.0.smithi164.stdout:290: oid 24 version 58 is already newer than 56 2024-04-26T01:08:37.524 INFO:tasks.rados.rados.0.smithi164.stdout:290: oid 24 updating version 58 to 59 2024-04-26T01:08:37.524 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 24 v 59 (ObjNum 128 snap 19 seq_num 128) dirty exists 2024-04-26T01:08:37.524 INFO:tasks.rados.rados.0.smithi164.stdout:290: left oid 24 (ObjNum 128 snap 19 seq_num 128) 2024-04-26T01:08:37.524 INFO:tasks.rados.rados.0.smithi164.stdout:291: finishing write tid 1 to smithi16428812-32 2024-04-26T01:08:37.524 INFO:tasks.rados.rados.0.smithi164.stdout:291: finishing write tid 2 to smithi16428812-32 2024-04-26T01:08:37.525 INFO:tasks.rados.rados.0.smithi164.stdout:291: finishing write tid 3 to smithi16428812-32 2024-04-26T01:08:37.525 INFO:tasks.rados.rados.0.smithi164.stdout:291: finishing write tid 4 to smithi16428812-32 2024-04-26T01:08:37.525 INFO:tasks.rados.rados.0.smithi164.stdout:291: finishing write tid 5 to smithi16428812-32 2024-04-26T01:08:37.525 INFO:tasks.rados.rados.0.smithi164.stdout:291: finishing write tid 6 to smithi16428812-32 2024-04-26T01:08:37.525 INFO:tasks.rados.rados.0.smithi164.stdout:291: oid 32 updating version 0 to 66 2024-04-26T01:08:37.525 INFO:tasks.rados.rados.0.smithi164.stdout:291: oid 32 version 66 is already newer than 65 2024-04-26T01:08:37.525 INFO:tasks.rados.rados.0.smithi164.stdout:291: oid 32 version 66 is already newer than 64 2024-04-26T01:08:37.525 INFO:tasks.rados.rados.0.smithi164.stdout:291: oid 32 version 66 is already newer than 63 2024-04-26T01:08:37.525 INFO:tasks.rados.rados.0.smithi164.stdout:291: oid 32 version 66 is already newer than 62 2024-04-26T01:08:37.525 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 32 v 66 (ObjNum 129 snap 19 seq_num 129) dirty exists 2024-04-26T01:08:37.525 INFO:tasks.rados.rados.0.smithi164.stdout:291: left oid 32 (ObjNum 129 snap 19 seq_num 129) 2024-04-26T01:08:37.525 INFO:tasks.rados.rados.0.smithi164.stdout:293: finishing write tid 1 to smithi16428812-36 2024-04-26T01:08:37.525 INFO:tasks.rados.rados.0.smithi164.stdout:293: finishing write tid 2 to smithi16428812-36 2024-04-26T01:08:37.525 INFO:tasks.rados.rados.0.smithi164.stdout:293: finishing write tid 3 to smithi16428812-36 2024-04-26T01:08:37.525 INFO:tasks.rados.rados.0.smithi164.stdout:293: finishing write tid 4 to smithi16428812-36 2024-04-26T01:08:37.526 INFO:tasks.rados.rados.0.smithi164.stdout:295: finishing write tid 1 to smithi16428812-48 2024-04-26T01:08:37.526 INFO:tasks.rados.rados.0.smithi164.stdout:295: finishing write tid 2 to smithi16428812-48 2024-04-26T01:08:37.526 INFO:tasks.rados.rados.0.smithi164.stdout:295: finishing write tid 3 to smithi16428812-48 2024-04-26T01:08:37.526 INFO:tasks.rados.rados.0.smithi164.stdout:295: finishing write tid 4 to smithi16428812-48 2024-04-26T01:08:37.526 INFO:tasks.rados.rados.0.smithi164.stdout:297: finishing write tid 1 to smithi16428812-35 2024-04-26T01:08:37.526 INFO:tasks.rados.rados.0.smithi164.stdout:297: finishing write tid 2 to smithi16428812-35 2024-04-26T01:08:37.526 INFO:tasks.rados.rados.0.smithi164.stdout:297: finishing write tid 3 to smithi16428812-35 2024-04-26T01:08:37.526 INFO:tasks.rados.rados.0.smithi164.stdout:297: finishing write tid 4 to smithi16428812-35 2024-04-26T01:08:37.526 INFO:tasks.rados.rados.0.smithi164.stdout:292: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-26T01:08:37.526 INFO:tasks.rados.rados.0.smithi164.stdout:288: finishing write tid 5 to smithi16428812-5 2024-04-26T01:08:37.527 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:37.516+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:37.580 INFO:tasks.rados.rados.0.smithi164.stdout:293: finishing write tid 5 to smithi16428812-36 2024-04-26T01:08:37.584 INFO:tasks.rados.rados.0.smithi164.stdout:293: finishing write tid 6 to smithi16428812-36 2024-04-26T01:08:37.584 INFO:tasks.rados.rados.0.smithi164.stdout:293: oid 36 updating version 0 to 97 2024-04-26T01:08:37.585 INFO:tasks.rados.rados.0.smithi164.stdout:293: oid 36 updating version 97 to 98 2024-04-26T01:08:37.585 INFO:tasks.rados.rados.0.smithi164.stdout:293: oid 36 updating version 98 to 99 2024-04-26T01:08:37.585 INFO:tasks.rados.rados.0.smithi164.stdout:293: oid 36 updating version 99 to 100 2024-04-26T01:08:37.585 INFO:tasks.rados.rados.0.smithi164.stdout:293: oid 36 version 100 is already newer than 96 2024-04-26T01:08:37.585 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 36 v 100 (ObjNum 130 snap 19 seq_num 130) dirty exists 2024-04-26T01:08:37.585 INFO:tasks.rados.rados.0.smithi164.stdout:293: left oid 36 (ObjNum 130 snap 19 seq_num 130) 2024-04-26T01:08:37.586 INFO:tasks.rados.rados.0.smithi164.stdout:297: finishing write tid 5 to smithi16428812-35 2024-04-26T01:08:37.593 INFO:tasks.rados.rados.0.smithi164.stdout:297: finishing write tid 6 to smithi16428812-35 2024-04-26T01:08:37.593 INFO:tasks.rados.rados.0.smithi164.stdout:297: oid 35 updating version 0 to 63 2024-04-26T01:08:37.593 INFO:tasks.rados.rados.0.smithi164.stdout:297: oid 35 version 63 is already newer than 61 2024-04-26T01:08:37.593 INFO:tasks.rados.rados.0.smithi164.stdout:297: oid 35 version 63 is already newer than 62 2024-04-26T01:08:37.593 INFO:tasks.rados.rados.0.smithi164.stdout:297: oid 35 updating version 63 to 64 2024-04-26T01:08:37.593 INFO:tasks.rados.rados.0.smithi164.stdout:297: oid 35 version 64 is already newer than 60 2024-04-26T01:08:37.593 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 35 v 64 (ObjNum 132 snap 19 seq_num 132) dirty exists 2024-04-26T01:08:37.594 INFO:tasks.rados.rados.0.smithi164.stdout:297: left oid 35 (ObjNum 132 snap 19 seq_num 132) 2024-04-26T01:08:37.599 INFO:tasks.rados.rados.0.smithi164.stdout:295: finishing write tid 5 to smithi16428812-48 2024-04-26T01:08:37.620 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:37.619+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:37.662 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:37.662 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:37.662 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:08:37.663 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:37.678 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:08:37.701 INFO:tasks.rados.rados.0.smithi164.stdout:288: finishing write tid 6 to smithi16428812-5 2024-04-26T01:08:37.701 INFO:tasks.rados.rados.0.smithi164.stdout:288: oid 5 updating version 0 to 28 2024-04-26T01:08:37.701 INFO:tasks.rados.rados.0.smithi164.stdout:288: oid 5 updating version 28 to 31 2024-04-26T01:08:37.701 INFO:tasks.rados.rados.0.smithi164.stdout:288: oid 5 version 31 is already newer than 30 2024-04-26T01:08:37.701 INFO:tasks.rados.rados.0.smithi164.stdout:288: oid 5 updating version 31 to 32 2024-04-26T01:08:37.702 INFO:tasks.rados.rados.0.smithi164.stdout:288: oid 5 version 32 is already newer than 29 2024-04-26T01:08:37.702 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 5 v 32 (ObjNum 127 snap 19 seq_num 127) dirty exists 2024-04-26T01:08:37.702 INFO:tasks.rados.rados.0.smithi164.stdout:288: left oid 5 (ObjNum 127 snap 19 seq_num 127) 2024-04-26T01:08:37.720 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:37.719+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:37.821 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:37.816+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:37.842 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:37.842 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:37.842 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:08:37.842 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:08:37.842 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:37.856 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:08:37.922 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:37.921+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:37.981 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:37.981 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:08:37.981 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:08:37.981 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:08:37.981 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.981 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:145 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'103, mlcod=25'98)", 2024-04-26T01:08:37.981 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:47.983085+0000", 2024-04-26T01:08:37.981 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 49.996515547000001, 2024-04-26T01:08:37.981 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.055794923000000003, 2024-04-26T01:08:37.981 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:37.982 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:37.982 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:37.982 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:37.982 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.982 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:37.982 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983085+0000", 2024-04-26T01:08:37.982 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:37.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.982 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.982 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:37.982 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983085+0000", 2024-04-26T01:08:37.982 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:37.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.983 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.983 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:37.983 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983086+0000", 2024-04-26T01:08:37.983 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9699999999999994e-07 2024-04-26T01:08:37.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.983 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.983 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:37.983 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983194+0000", 2024-04-26T01:08:37.983 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107569 2024-04-26T01:08:37.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.983 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.983 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:37.983 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037255+0000", 2024-04-26T01:08:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.054061302999999998 2024-04-26T01:08:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037268+0000", 2024-04-26T01:08:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3477e-05 2024-04-26T01:08:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037378+0000", 2024-04-26T01:08:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000110089 2024-04-26T01:08:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:37.985 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037414+0000", 2024-04-26T01:08:37.985 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5123000000000002e-05 2024-04-26T01:08:37.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.985 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.985 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:37.985 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.038790+0000", 2024-04-26T01:08:37.985 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0013768820000000001 2024-04-26T01:08:37.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.985 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.985 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:37.985 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.038880+0000", 2024-04-26T01:08:37.985 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9482999999999997e-05 2024-04-26T01:08:37.985 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:37.986 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:37.986 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:37.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.986 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.986 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:24 3.es0 3:7b1f3e32:::smithi16428812-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:37.986 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.309434+0000", 2024-04-26T01:08:37.986 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.670166518000002, 2024-04-26T01:08:37.986 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.019012134999999999, 2024-04-26T01:08:37.986 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:37.986 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:37.986 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:37.986 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:37.987 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:37.987 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:37.987 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 24 2024-04-26T01:08:37.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.987 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:37.987 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.987 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:37.987 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309434+0000", 2024-04-26T01:08:37.987 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:37.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.987 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.987 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:37.987 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309434+0000", 2024-04-26T01:08:37.987 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:37.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.988 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.988 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:37.988 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309435+0000", 2024-04-26T01:08:37.988 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.3e-07 2024-04-26T01:08:37.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.988 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.988 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:37.988 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309439+0000", 2024-04-26T01:08:37.988 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8530000000000002e-06 2024-04-26T01:08:37.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.988 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.989 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:37.989 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309448+0000", 2024-04-26T01:08:37.989 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9879999999999993e-06 2024-04-26T01:08:37.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.989 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.989 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.309454+0000", 2024-04-26T01:08:37.989 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.6630000000000004e-06 2024-04-26T01:08:37.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.989 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:37.989 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312756+0000", 2024-04-26T01:08:37.989 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003301712 2024-04-26T01:08:37.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.990 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.990 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:37.990 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.312870+0000", 2024-04-26T01:08:37.990 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000113983 2024-04-26T01:08:37.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.990 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.990 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:37.990 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.324091+0000", 2024-04-26T01:08:37.990 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011221419 2024-04-26T01:08:37.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.990 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.990 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:37.991 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.324213+0000", 2024-04-26T01:08:37.991 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012186200000000001 2024-04-26T01:08:37.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.991 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.991 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:37.991 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.328446+0000", 2024-04-26T01:08:37.991 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0042329250000000002 2024-04-26T01:08:37.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:37.991 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:37.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:37.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.991 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.991 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpRead(3.2s1 33/30 ECSubRead(tid=163, to_read={3:410d186f:::smithi16428812-11:head=0,4096,0}, subchunks={3:410d186f:::smithi16428812-11:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:37.991 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.340790+0000", 2024-04-26T01:08:37.992 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.638810550000002, 2024-04-26T01:08:37.992 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.047962901000000002, 2024-04-26T01:08:37.992 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:37.992 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:37.992 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:37.992 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:37.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.992 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:37.992 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340790+0000", 2024-04-26T01:08:37.992 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:37.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.993 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:37.993 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340790+0000", 2024-04-26T01:08:37.993 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:37.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.993 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.993 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:37.993 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340791+0000", 2024-04-26T01:08:37.993 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.3300000000000001e-07 2024-04-26T01:08:37.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.993 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.993 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:37.993 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340801+0000", 2024-04-26T01:08:37.993 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0308e-05 2024-04-26T01:08:37.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.994 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.994 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:37.994 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388316+0000", 2024-04-26T01:08:37.994 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.047514702999999998 2024-04-26T01:08:37.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.994 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.994 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.994 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388324+0000", 2024-04-26T01:08:37.994 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.3529999999999995e-06 2024-04-26T01:08:37.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.994 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.994 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:37.995 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388439+0000", 2024-04-26T01:08:37.995 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000114657 2024-04-26T01:08:37.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.995 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.995 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:37.995 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388753+0000", 2024-04-26T01:08:37.995 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00031404700000000002 2024-04-26T01:08:37.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:37.995 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:37.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:37.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.995 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.996 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:62 3.es0 3:7eef64f7:::smithi16428812-21:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:37.996 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.361875+0000", 2024-04-26T01:08:37.996 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.617725475999997, 2024-04-26T01:08:37.996 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.111184554, 2024-04-26T01:08:37.996 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:37.996 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:37.996 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:37.996 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:37.996 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:37.996 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:37.996 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 62 2024-04-26T01:08:37.996 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.997 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:37.997 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.997 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:37.997 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361875+0000", 2024-04-26T01:08:37.997 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:37.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.997 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.997 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:37.997 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361875+0000", 2024-04-26T01:08:37.997 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:37.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.997 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.997 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:37.997 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361876+0000", 2024-04-26T01:08:37.998 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5899999999999995e-07 2024-04-26T01:08:37.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.998 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.998 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:37.998 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361880+0000", 2024-04-26T01:08:37.998 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8739999999999997e-06 2024-04-26T01:08:37.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.998 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.998 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:37.998 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361887+0000", 2024-04-26T01:08:37.998 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4839999999999998e-06 2024-04-26T01:08:37.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.998 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.999 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:37.999 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361892+0000", 2024-04-26T01:08:37.999 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.437e-06 2024-04-26T01:08:37.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.999 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.999 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:37.999 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.363374+0000", 2024-04-26T01:08:37.999 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001482693 2024-04-26T01:08:37.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:37.999 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:37.999 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:37.999 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.363442+0000", 2024-04-26T01:08:37.999 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7448999999999997e-05 2024-04-26T01:08:37.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.000 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.000 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.000 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.363805+0000", 2024-04-26T01:08:38.000 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00036289900000000002 2024-04-26T01:08:38.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.000 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.000 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.000 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.366199+0000", 2024-04-26T01:08:38.000 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.002394654 2024-04-26T01:08:38.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.000 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.000 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:38.000 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472968+0000", 2024-04-26T01:08:38.000 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.106769087 2024-04-26T01:08:38.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.001 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.001 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473060+0000", 2024-04-26T01:08:38.001 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.1218000000000007e-05 2024-04-26T01:08:38.001 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.001 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.001 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.001 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:63 3.es0 3:7eef64f7:::smithi16428812-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:38.001 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.361913+0000", 2024-04-26T01:08:38.001 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.617687531999998, 2024-04-26T01:08:38.001 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.11409399200000001, 2024-04-26T01:08:38.002 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.002 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.002 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:38.002 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:38.002 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:38.002 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:38.002 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 63 2024-04-26T01:08:38.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.002 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.002 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.002 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.002 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361913+0000", 2024-04-26T01:08:38.002 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.003 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.003 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361913+0000", 2024-04-26T01:08:38.003 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.003 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.003 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361914+0000", 2024-04-26T01:08:38.003 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.5700000000000002e-07 2024-04-26T01:08:38.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.003 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.003 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361917+0000", 2024-04-26T01:08:38.004 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0979999999999998e-06 2024-04-26T01:08:38.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.004 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.004 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.004 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361923+0000", 2024-04-26T01:08:38.004 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9719999999999997e-06 2024-04-26T01:08:38.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.004 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.004 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.004 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361927+0000", 2024-04-26T01:08:38.004 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.5070000000000002e-06 2024-04-26T01:08:38.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.004 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.004 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.004 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.364059+0000", 2024-04-26T01:08:38.005 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.002132278 2024-04-26T01:08:38.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.005 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:38.005 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.364206+0000", 2024-04-26T01:08:38.005 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014703700000000001 2024-04-26T01:08:38.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.005 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.005 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473123+0000", 2024-04-26T01:08:38.005 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.108916164 2024-04-26T01:08:38.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.006 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:38.006 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473211+0000", 2024-04-26T01:08:38.006 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8381999999999995e-05 2024-04-26T01:08:38.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.006 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.006 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.006 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.476007+0000", 2024-04-26T01:08:38.006 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0027959970000000002 2024-04-26T01:08:38.006 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.006 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.006 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.006 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.006 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWriteReply(3.es0 33/30 ECSubWriteReply(tid=5, last_complete=33'4, committed=1, applied=1))", 2024-04-26T01:08:38.007 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.365083+0000", 2024-04-26T01:08:38.007 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.614517050000003, 2024-04-26T01:08:38.007 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.107988984, 2024-04-26T01:08:38.007 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.007 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.007 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.007 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.007 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.007 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365083+0000", 2024-04-26T01:08:38.007 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.007 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.008 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365083+0000", 2024-04-26T01:08:38.008 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.008 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.008 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.008 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365084+0000", 2024-04-26T01:08:38.008 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.9299999999999999e-07 2024-04-26T01:08:38.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.008 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.008 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.008 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365086+0000", 2024-04-26T01:08:38.009 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5239999999999999e-06 2024-04-26T01:08:38.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.009 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.009 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472672+0000", 2024-04-26T01:08:38.009 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.10758553899999999 2024-04-26T01:08:38.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.010 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.010 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.010 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472680+0000", 2024-04-26T01:08:38.010 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.6820000000000003e-06 2024-04-26T01:08:38.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.010 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.010 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.010 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472900+0000", 2024-04-26T01:08:38.010 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00022041899999999999 2024-04-26T01:08:38.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.010 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.010 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.011 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473072+0000", 2024-04-26T01:08:38.011 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017242700000000001 2024-04-26T01:08:38.011 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.011 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.011 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.011 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.2s1 33/30 ECSubWrite(tid=165, reqid=client.4521.0:80, at_version=33'6, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:08:38.011 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.391943+0000", 2024-04-26T01:08:38.011 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.587657372000002, 2024-04-26T01:08:38.011 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.042848432999999998, 2024-04-26T01:08:38.011 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.011 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.012 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.012 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.012 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.012 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.012 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391943+0000", 2024-04-26T01:08:38.012 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.012 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.012 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.012 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391943+0000", 2024-04-26T01:08:38.012 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.012 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.012 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.013 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391944+0000", 2024-04-26T01:08:38.013 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8199999999999998e-07 2024-04-26T01:08:38.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.013 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.013 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.013 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391961+0000", 2024-04-26T01:08:38.013 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6546999999999999e-05 2024-04-26T01:08:38.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.013 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.013 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.013 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.430981+0000", 2024-04-26T01:08:38.013 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.039020817999999999 2024-04-26T01:08:38.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.013 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.014 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.014 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.430991+0000", 2024-04-26T01:08:38.014 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7059999999999999e-06 2024-04-26T01:08:38.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.014 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.014 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.014 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.431059+0000", 2024-04-26T01:08:38.014 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.8198e-05 2024-04-26T01:08:38.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.014 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.014 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.014 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.431067+0000", 2024-04-26T01:08:38.014 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.8930000000000005e-06 2024-04-26T01:08:38.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.015 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.015 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.434752+0000", 2024-04-26T01:08:38.015 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0036849249999999999 2024-04-26T01:08:38.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.015 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.015 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.434792+0000", 2024-04-26T01:08:38.015 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.9464e-05 2024-04-26T01:08:38.015 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.015 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.015 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.016 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.016 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.3s4 33/30 ECSubWrite(tid=170, reqid=client.4521.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=33'2))", 2024-04-26T01:08:38.016 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.441271+0000", 2024-04-26T01:08:38.016 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.538329593999997, 2024-04-26T01:08:38.016 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.063989213000000003, 2024-04-26T01:08:38.016 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.016 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.016 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.016 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.016 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.016 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.016 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441271+0000", 2024-04-26T01:08:38.016 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.017 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.017 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.017 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441271+0000", 2024-04-26T01:08:38.017 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.017 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.017 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.017 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441272+0000", 2024-04-26T01:08:38.017 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1650000000000001e-06 2024-04-26T01:08:38.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.017 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.017 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.017 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441286+0000", 2024-04-26T01:08:38.018 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3771000000000001e-05 2024-04-26T01:08:38.018 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.018 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.018 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.018 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504639+0000", 2024-04-26T01:08:38.018 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.063352787999999993 2024-04-26T01:08:38.018 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.018 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.018 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.019 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504647+0000", 2024-04-26T01:08:38.019 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9389999999999997e-06 2024-04-26T01:08:38.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.019 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.019 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.019 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504715+0000", 2024-04-26T01:08:38.019 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7926e-05 2024-04-26T01:08:38.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.019 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.019 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.019 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504722+0000", 2024-04-26T01:08:38.019 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9449999999999993e-06 2024-04-26T01:08:38.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.020 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.020 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.020 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.505203+0000", 2024-04-26T01:08:38.020 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00048092399999999998 2024-04-26T01:08:38.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.020 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.020 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.020 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.505260+0000", 2024-04-26T01:08:38.020 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.6755000000000001e-05 2024-04-26T01:08:38.020 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.020 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.020 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.020 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.021 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:233 3.es0 3:7b1f3e32:::smithi16428812-8:head [copy-from ver 4] snapc 3=[2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-26T01:08:38.021 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:02.464122+0000", 2024-04-26T01:08:38.021 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 35.515478952000002, 2024-04-26T01:08:38.021 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.66639822699999995, 2024-04-26T01:08:38.021 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.021 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.021 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:38.021 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:38.021 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:38.021 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:38.021 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 233 2024-04-26T01:08:38.021 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.022 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.022 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.022 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464122+0000", 2024-04-26T01:08:38.022 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.022 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.022 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464122+0000", 2024-04-26T01:08:38.022 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.023 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.023 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464125+0000", 2024-04-26T01:08:38.023 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0939999999999999e-06 2024-04-26T01:08:38.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.023 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.023 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.023 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464136+0000", 2024-04-26T01:08:38.023 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1790000000000001e-05 2024-04-26T01:08:38.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.023 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.023 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.023 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464160+0000", 2024-04-26T01:08:38.023 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3683e-05 2024-04-26T01:08:38.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.024 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.024 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464174+0000", 2024-04-26T01:08:38.024 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.382e-05 2024-04-26T01:08:38.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.024 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.024 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464324+0000", 2024-04-26T01:08:38.024 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014991900000000001 2024-04-26T01:08:38.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.024 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:38.024 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464485+0000", 2024-04-26T01:08:38.025 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016099500000000001 2024-04-26T01:08:38.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.025 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.025 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.025 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480641+0000", 2024-04-26T01:08:38.025 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016155748000000001 2024-04-26T01:08:38.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.025 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.025 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.025 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.485807+0000", 2024-04-26T01:08:38.025 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0051663380000000004 2024-04-26T01:08:38.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.025 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.130287+0000", 2024-04-26T01:08:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.64448033500000002 2024-04-26T01:08:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.130520+0000", 2024-04-26T01:08:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00023250500000000001 2024-04-26T01:08:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:234 3.es0 3:7b1f3e32:::smithi16428812-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-04-26T01:08:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:02.464220+0000", 2024-04-26T01:08:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 35.515380608999997, 2024-04-26T01:08:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.019428925, 2024-04-26T01:08:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 234 2024-04-26T01:08:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464220+0000", 2024-04-26T01:08:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464220+0000", 2024-04-26T01:08:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464221+0000", 2024-04-26T01:08:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4500000000000001e-06 2024-04-26T01:08:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464229+0000", 2024-04-26T01:08:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.6229999999999997e-06 2024-04-26T01:08:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464250+0000", 2024-04-26T01:08:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1163e-05 2024-04-26T01:08:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464267+0000", 2024-04-26T01:08:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.668e-05 2024-04-26T01:08:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464815+0000", 2024-04-26T01:08:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00054803000000000002 2024-04-26T01:08:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for blocked object", 2024-04-26T01:08:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464939+0000", 2024-04-26T01:08:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000124385 2024-04-26T01:08:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.483264+0000", 2024-04-26T01:08:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.018324427000000001 2024-04-26T01:08:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.483429+0000", 2024-04-26T01:08:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000165749 2024-04-26T01:08:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.483649+0000", 2024-04-26T01:08:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00021941800000000001 2024-04-26T01:08:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:257 3.es0 3:7b1f3e32:::smithi16428812-8:head [append 0~0] snapc 4=[4,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:03.446919+0000", 2024-04-26T01:08:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 34.532681642999997, 2024-04-26T01:08:38.033 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.84360518900000003, 2024-04-26T01:08:38.033 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.033 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.033 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:38.033 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:38.033 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:38.033 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:38.033 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 257 2024-04-26T01:08:38.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.033 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.033 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.033 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.033 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446919+0000", 2024-04-26T01:08:38.033 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.034 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.034 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.034 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446919+0000", 2024-04-26T01:08:38.034 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.034 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.034 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.034 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446920+0000", 2024-04-26T01:08:38.034 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4700000000000001e-07 2024-04-26T01:08:38.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.034 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.034 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.034 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446922+0000", 2024-04-26T01:08:38.035 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6800000000000002e-06 2024-04-26T01:08:38.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.035 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.035 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.035 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446931+0000", 2024-04-26T01:08:38.035 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8659999999999995e-06 2024-04-26T01:08:38.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.035 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.035 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.035 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446941+0000", 2024-04-26T01:08:38.035 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.5540000000000003e-06 2024-04-26T01:08:38.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.035 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.035 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.036 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.458756+0000", 2024-04-26T01:08:38.036 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011815275 2024-04-26T01:08:38.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.036 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.036 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:38.036 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.459005+0000", 2024-04-26T01:08:38.036 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000249418 2024-04-26T01:08:38.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.036 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.036 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.036 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.460129+0000", 2024-04-26T01:08:38.036 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001123768 2024-04-26T01:08:38.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.037 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.037 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.037 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461504+0000", 2024-04-26T01:08:38.037 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001375079 2024-04-26T01:08:38.037 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.037 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.037 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:38.037 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.290324+0000", 2024-04-26T01:08:38.037 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.82881998700000004 2024-04-26T01:08:38.037 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.037 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.037 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.037 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.290524+0000", 2024-04-26T01:08:38.037 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000199815 2024-04-26T01:08:38.038 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.038 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.038 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.038 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.038 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.038 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/30 ECSubWriteReply(tid=25, last_complete=37'19, committed=1, applied=1))", 2024-04-26T01:08:38.038 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:03.461054+0000", 2024-04-26T01:08:38.038 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 34.518546655000002, 2024-04-26T01:08:38.038 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.83058926899999996, 2024-04-26T01:08:38.038 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.038 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.038 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.038 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.039 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.039 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.039 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461054+0000", 2024-04-26T01:08:38.039 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.039 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.039 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.039 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461054+0000", 2024-04-26T01:08:38.040 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.040 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.040 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.040 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461056+0000", 2024-04-26T01:08:38.040 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.906e-06 2024-04-26T01:08:38.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.040 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.040 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.040 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461069+0000", 2024-04-26T01:08:38.040 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.295e-05 2024-04-26T01:08:38.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.041 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.041 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.041 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.289919+0000", 2024-04-26T01:08:38.041 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.82884987300000001 2024-04-26T01:08:38.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.041 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.041 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.041 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.289933+0000", 2024-04-26T01:08:38.041 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4430000000000001e-05 2024-04-26T01:08:38.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.041 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.041 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.041 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.290082+0000", 2024-04-26T01:08:38.042 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014943899999999999 2024-04-26T01:08:38.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.042 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.042 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.042 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.291643+0000", 2024-04-26T01:08:38.042 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001560671 2024-04-26T01:08:38.042 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.042 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.042 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.042 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.042 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:385 3.fs1 3.db40cadf (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:38.042 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.452224+0000", 2024-04-26T01:08:38.043 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 28.527376904, 2024-04-26T01:08:38.043 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95265389199999995, 2024-04-26T01:08:38.043 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.043 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.043 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.043 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:38.043 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:38.043 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:38.043 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 385 2024-04-26T01:08:38.043 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.043 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.043 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.043 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.043 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452224+0000", 2024-04-26T01:08:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452224+0000", 2024-04-26T01:08:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452227+0000", 2024-04-26T01:08:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6830000000000001e-06 2024-04-26T01:08:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452242+0000", 2024-04-26T01:08:38.045 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4493e-05 2024-04-26T01:08:38.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.045 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.045 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.045 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452274+0000", 2024-04-26T01:08:38.045 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1806e-05 2024-04-26T01:08:38.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.045 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.045 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.045 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452300+0000", 2024-04-26T01:08:38.045 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6092999999999999e-05 2024-04-26T01:08:38.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.045 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.045 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.046 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452469+0000", 2024-04-26T01:08:38.046 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000169629 2024-04-26T01:08:38.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.046 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:08:38.046 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452475+0000", 2024-04-26T01:08:38.046 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.604e-06 2024-04-26T01:08:38.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.046 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.046 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.404853+0000", 2024-04-26T01:08:38.046 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95237828400000002 2024-04-26T01:08:38.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.404878+0000", 2024-04-26T01:08:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4300000000000001e-05 2024-04-26T01:08:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:386 3.fs1 3.db40cadf (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.452348+0000", 2024-04-26T01:08:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 28.527252701999998, 2024-04-26T01:08:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95267636899999997, 2024-04-26T01:08:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 386 2024-04-26T01:08:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452348+0000", 2024-04-26T01:08:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452348+0000", 2024-04-26T01:08:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452349+0000", 2024-04-26T01:08:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.533e-06 2024-04-26T01:08:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452359+0000", 2024-04-26T01:08:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.5070000000000006e-06 2024-04-26T01:08:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452381+0000", 2024-04-26T01:08:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2306e-05 2024-04-26T01:08:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452398+0000", 2024-04-26T01:08:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6456e-05 2024-04-26T01:08:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452605+0000", 2024-04-26T01:08:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00020750900000000001 2024-04-26T01:08:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:08:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452610+0000", 2024-04-26T01:08:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3569999999999997e-06 2024-04-26T01:08:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.404988+0000", 2024-04-26T01:08:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95237776100000004 2024-04-26T01:08:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405024+0000", 2024-04-26T01:08:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5939999999999998e-05 2024-04-26T01:08:38.055 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.055 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.055 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.055 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:387 3.fs1 3.db40cadf (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:38.055 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.452423+0000", 2024-04-26T01:08:38.055 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 28.527177296000001, 2024-04-26T01:08:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95294480000000004, 2024-04-26T01:08:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 387 2024-04-26T01:08:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452423+0000", 2024-04-26T01:08:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452423+0000", 2024-04-26T01:08:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452425+0000", 2024-04-26T01:08:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.373e-06 2024-04-26T01:08:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.060 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452435+0000", 2024-04-26T01:08:38.060 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0149999999999999e-05 2024-04-26T01:08:38.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.060 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.060 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452447+0000", 2024-04-26T01:08:38.060 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.199e-05 2024-04-26T01:08:38.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.060 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.060 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452461+0000", 2024-04-26T01:08:38.060 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4503e-05 2024-04-26T01:08:38.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.061 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.061 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452685+0000", 2024-04-26T01:08:38.061 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00022367200000000001 2024-04-26T01:08:38.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.061 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:08:38.061 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452687+0000", 2024-04-26T01:08:38.061 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.446e-06 2024-04-26T01:08:38.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.061 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.061 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405319+0000", 2024-04-26T01:08:38.061 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.952631279 2024-04-26T01:08:38.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.062 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.062 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405368+0000", 2024-04-26T01:08:38.062 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9387e-05 2024-04-26T01:08:38.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.062 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.062 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:393 3.fs1 3.94f9353f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:38.062 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.464778+0000", 2024-04-26T01:08:38.062 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 28.514822767999998, 2024-04-26T01:08:38.062 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94080343600000005, 2024-04-26T01:08:38.063 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.063 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.063 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.063 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:38.063 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:38.063 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:38.063 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 393 2024-04-26T01:08:38.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.063 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464778+0000", 2024-04-26T01:08:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464778+0000", 2024-04-26T01:08:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464780+0000", 2024-04-26T01:08:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2460000000000002e-06 2024-04-26T01:08:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.065 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.065 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.065 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464791+0000", 2024-04-26T01:08:38.065 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.145e-05 2024-04-26T01:08:38.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.065 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.065 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.065 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464823+0000", 2024-04-26T01:08:38.065 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1157e-05 2024-04-26T01:08:38.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.065 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.065 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.065 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464837+0000", 2024-04-26T01:08:38.065 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.488e-05 2024-04-26T01:08:38.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.066 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.066 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.066 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464985+0000", 2024-04-26T01:08:38.066 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000147999 2024-04-26T01:08:38.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.066 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.066 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:08:38.066 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464991+0000", 2024-04-26T01:08:38.066 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3329999999999996e-06 2024-04-26T01:08:38.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.066 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.067 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.067 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405522+0000", 2024-04-26T01:08:38.067 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94053098899999998 2024-04-26T01:08:38.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.067 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.067 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.067 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405581+0000", 2024-04-26T01:08:38.067 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9382000000000002e-05 2024-04-26T01:08:38.067 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.067 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.067 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.067 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.067 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:394 3.fs1 3.94f9353f (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:38.068 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.464870+0000", 2024-04-26T01:08:38.068 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 28.514730491999998, 2024-04-26T01:08:38.068 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94122545000000002, 2024-04-26T01:08:38.068 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.068 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.068 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.068 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:38.068 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:38.068 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:38.068 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 394 2024-04-26T01:08:38.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.068 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.068 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464870+0000", 2024-04-26T01:08:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464870+0000", 2024-04-26T01:08:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464871+0000", 2024-04-26T01:08:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1230000000000001e-06 2024-04-26T01:08:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.070 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.070 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464875+0000", 2024-04-26T01:08:38.070 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.25e-06 2024-04-26T01:08:38.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.070 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.070 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.070 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464895+0000", 2024-04-26T01:08:38.070 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9332999999999999e-05 2024-04-26T01:08:38.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.070 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.070 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.070 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464904+0000", 2024-04-26T01:08:38.070 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9230000000000008e-06 2024-04-26T01:08:38.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.071 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.071 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.071 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.465051+0000", 2024-04-26T01:08:38.071 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000147186 2024-04-26T01:08:38.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.071 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.071 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:08:38.071 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.465054+0000", 2024-04-26T01:08:38.071 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7530000000000002e-06 2024-04-26T01:08:38.072 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.072 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.072 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.072 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.406060+0000", 2024-04-26T01:08:38.072 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94100597900000005 2024-04-26T01:08:38.072 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.072 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.072 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.073 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.406095+0000", 2024-04-26T01:08:38.073 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5902999999999998e-05 2024-04-26T01:08:38.073 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.073 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.073 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.073 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.073 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpRead(3.fs1 58/44 ECSubRead(tid=311, to_read={3:fa21e776:::smithi16428812-18:head=0,4096,0}, subchunks={3:fa21e776:::smithi16428812-18:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:38.073 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:24.214139+0000", 2024-04-26T01:08:38.073 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.765461351000001, 2024-04-26T01:08:38.073 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.068278328999999999, 2024-04-26T01:08:38.073 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.073 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.073 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.074 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.074 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.074 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.074 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.214139+0000", 2024-04-26T01:08:38.074 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.074 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.074 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.074 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.214139+0000", 2024-04-26T01:08:38.074 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.074 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.075 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.075 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.214141+0000", 2024-04-26T01:08:38.075 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.674e-06 2024-04-26T01:08:38.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.075 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.075 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.075 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.214161+0000", 2024-04-26T01:08:38.075 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0349000000000001e-05 2024-04-26T01:08:38.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.075 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.075 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.075 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.281949+0000", 2024-04-26T01:08:38.075 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.067788082999999999 2024-04-26T01:08:38.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.076 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.076 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.281978+0000", 2024-04-26T01:08:38.076 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8646999999999999e-05 2024-04-26T01:08:38.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.076 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.076 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.282180+0000", 2024-04-26T01:08:38.076 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00020251500000000001 2024-04-26T01:08:38.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.076 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.076 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.282417+0000", 2024-04-26T01:08:38.077 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000237061 2024-04-26T01:08:38.077 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.077 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.077 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.077 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:691 3.es0 3:7eef64f7:::smithi16428812-21:head [read 0~2719744,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-26T01:08:38.077 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:30.194502+0000", 2024-04-26T01:08:38.077 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 7.7850987859999998, 2024-04-26T01:08:38.077 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.018755170000000002, 2024-04-26T01:08:38.077 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.077 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.077 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:38.078 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:38.078 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:38.078 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:38.078 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 691 2024-04-26T01:08:38.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.078 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.078 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.078 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.078 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194502+0000", 2024-04-26T01:08:38.078 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.078 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.078 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.078 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194502+0000", 2024-04-26T01:08:38.079 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.079 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.079 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194503+0000", 2024-04-26T01:08:38.079 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0529999999999999e-06 2024-04-26T01:08:38.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.079 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.079 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194508+0000", 2024-04-26T01:08:38.079 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.6969999999999998e-06 2024-04-26T01:08:38.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.079 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.080 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194522+0000", 2024-04-26T01:08:38.080 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3485999999999999e-05 2024-04-26T01:08:38.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.080 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.080 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.080 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194542+0000", 2024-04-26T01:08:38.080 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9987e-05 2024-04-26T01:08:38.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.080 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.080 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.080 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194856+0000", 2024-04-26T01:08:38.080 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00031445699999999997 2024-04-26T01:08:38.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.080 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.081 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:38.081 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194867+0000", 2024-04-26T01:08:38.081 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.061e-05 2024-04-26T01:08:38.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.081 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.081 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.201006+0000", 2024-04-26T01:08:38.081 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061384990000000004 2024-04-26T01:08:38.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.081 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:38.081 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.201101+0000", 2024-04-26T01:08:38.081 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.5365000000000006e-05 2024-04-26T01:08:38.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.082 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.082 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.082 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.213257+0000", 2024-04-26T01:08:38.082 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012156016 2024-04-26T01:08:38.082 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.082 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.082 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.082 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.082 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:692 3.es0 3:7eef64f7:::smithi16428812-21:head [read 0~2719744,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-26T01:08:38.082 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:30.194619+0000", 2024-04-26T01:08:38.082 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 7.7849810569999995, 2024-04-26T01:08:38.082 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.021611364000000001, 2024-04-26T01:08:38.082 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.083 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.083 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:38.083 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:38.083 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:38.083 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:38.083 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 692 2024-04-26T01:08:38.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.083 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.083 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.083 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194619+0000", 2024-04-26T01:08:38.083 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.083 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.084 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194619+0000", 2024-04-26T01:08:38.084 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.084 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.084 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194620+0000", 2024-04-26T01:08:38.084 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7300000000000004e-07 2024-04-26T01:08:38.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.085 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.085 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194641+0000", 2024-04-26T01:08:38.085 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0910000000000001e-05 2024-04-26T01:08:38.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.085 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.085 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194653+0000", 2024-04-26T01:08:38.085 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1663e-05 2024-04-26T01:08:38.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.085 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.085 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194661+0000", 2024-04-26T01:08:38.085 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.49e-06 2024-04-26T01:08:38.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.086 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.086 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194940+0000", 2024-04-26T01:08:38.086 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00027803799999999998 2024-04-26T01:08:38.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.086 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:38.086 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194956+0000", 2024-04-26T01:08:38.087 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6702999999999999e-05 2024-04-26T01:08:38.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.087 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.087 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.201697+0000", 2024-04-26T01:08:38.087 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0067405629999999998 2024-04-26T01:08:38.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.087 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:38.087 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.201779+0000", 2024-04-26T01:08:38.087 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2678999999999999e-05 2024-04-26T01:08:38.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.088 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.088 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.216231+0000", 2024-04-26T01:08:38.088 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014451345000000001 2024-04-26T01:08:38.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.088 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.088 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.088 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:38.089 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-26T01:08:38.091 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:38.085+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:38.091 INFO:tasks.rados.rados.0.smithi164.stdout:295: finishing write tid 6 to smithi16428812-48 2024-04-26T01:08:38.092 INFO:tasks.rados.rados.0.smithi164.stdout:295: oid 48 updating version 0 to 61 2024-04-26T01:08:38.092 INFO:tasks.rados.rados.0.smithi164.stdout:295: oid 48 updating version 61 to 63 2024-04-26T01:08:38.092 INFO:tasks.rados.rados.0.smithi164.stdout:295: oid 48 version 63 is already newer than 60 2024-04-26T01:08:38.092 INFO:tasks.rados.rados.0.smithi164.stdout:295: oid 48 version 63 is already newer than 62 2024-04-26T01:08:38.092 INFO:tasks.rados.rados.0.smithi164.stdout:295: oid 48 version 63 is already newer than 59 2024-04-26T01:08:38.093 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 48 v 63 (ObjNum 131 snap 19 seq_num 131) dirty exists 2024-04-26T01:08:38.093 INFO:tasks.rados.rados.0.smithi164.stdout:295: left oid 48 (ObjNum 131 snap 19 seq_num 131) 2024-04-26T01:08:38.093 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:08:38.191 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:38.188+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:38.219 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:08:38.219 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:38.234 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:08:38.238 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.7 to 1 (10000) 2024-04-26T01:08:38.241 INFO:tasks.rados.rados.0.smithi164.stdout:286: done (9 left) 2024-04-26T01:08:38.241 INFO:tasks.rados.rados.0.smithi164.stdout:287: done (8 left) 2024-04-26T01:08:38.241 INFO:tasks.rados.rados.0.smithi164.stdout:288: done (7 left) 2024-04-26T01:08:38.241 INFO:tasks.rados.rados.0.smithi164.stdout:290: done (6 left) 2024-04-26T01:08:38.241 INFO:tasks.rados.rados.0.smithi164.stdout:291: done (5 left) 2024-04-26T01:08:38.241 INFO:tasks.rados.rados.0.smithi164.stdout:292: done (4 left) 2024-04-26T01:08:38.241 INFO:tasks.rados.rados.0.smithi164.stdout:293: done (3 left) 2024-04-26T01:08:38.241 INFO:tasks.rados.rados.0.smithi164.stdout:295: done (2 left) 2024-04-26T01:08:38.242 INFO:tasks.rados.rados.0.smithi164.stdout:297: done (1 left) 2024-04-26T01:08:38.242 INFO:tasks.rados.rados.0.smithi164.stdout:298: done (0 left) 2024-04-26T01:08:38.242 INFO:tasks.rados.rados.0.smithi164.stdout:299: read oid 17 snap -1 2024-04-26T01:08:38.242 INFO:tasks.rados.rados.0.smithi164.stdout:299: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-26T01:08:38.242 INFO:tasks.rados.rados.0.smithi164.stdout:300: delete oid 2 current snap is 20 2024-04-26T01:08:38.250 INFO:tasks.rados.rados.0.smithi164.stdout:300: done (1 left) 2024-04-26T01:08:38.250 INFO:tasks.rados.rados.0.smithi164.stdout:301: setattr oid 37 current snap is 20 2024-04-26T01:08:38.252 INFO:tasks.rados.rados.0.smithi164.stdout:302: copy_from oid 3 from oid 40 current snap is 20 2024-04-26T01:08:38.252 INFO:tasks.rados.rados.0.smithi164.stdout:303: delete oid 10 current snap is 20 2024-04-26T01:08:38.256 INFO:tasks.rados.rados.0.smithi164.stdout:299: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-26T01:08:38.264 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-04-26T01:08:38.275 INFO:tasks.rados.rados.0.smithi164.stdout:299: done (3 left) 2024-04-26T01:08:38.275 INFO:tasks.rados.rados.0.smithi164.stdout:303: done (2 left) 2024-04-26T01:08:38.276 INFO:tasks.rados.rados.0.smithi164.stdout:304: copy_from oid 28 from oid 49 current snap is 20 2024-04-26T01:08:38.276 INFO:tasks.rados.rados.0.smithi164.stdout:302: finishing copy_from racing read to smithi16428812-3 2024-04-26T01:08:38.276 INFO:tasks.rados.rados.0.smithi164.stdout:302: finishing copy_from to smithi16428812-3 2024-04-26T01:08:38.276 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 3 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-26T01:08:38.276 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 37 v 20 (ObjNum 120 snap 15 seq_num 120) dirty exists 2024-04-26T01:08:38.276 INFO:tasks.rados.rados.0.smithi164.stdout:301: done (2 left) 2024-04-26T01:08:38.276 INFO:tasks.rados.rados.0.smithi164.stdout:302: done (1 left) 2024-04-26T01:08:38.276 INFO:tasks.rados.rados.0.smithi164.stdout:305: rollback oid 2 current snap is 20 2024-04-26T01:08:38.277 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 2 to 19 2024-04-26T01:08:38.277 INFO:tasks.rados.rados.0.smithi164.stdout:306: snap_create 2024-04-26T01:08:38.279 INFO:tasks.rados.rados.0.smithi164.stdout:305: finishing rollback tid 1 to smithi16428812-2 2024-04-26T01:08:38.281 INFO:tasks.rados.rados.0.smithi164.stdout:305: finishing rollback tid 2 to smithi16428812-2 2024-04-26T01:08:38.281 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 2 v 29 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-26T01:08:38.291 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:38.289+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:38.308 INFO:tasks.rados.rados.0.smithi164.stdout:304: finishing copy_from racing read to smithi16428812-28 2024-04-26T01:08:38.311 INFO:tasks.rados.rados.0.smithi164.stdout:304: finishing copy_from to smithi16428812-28 2024-04-26T01:08:38.311 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 28 v 62 (ObjNum 108 snap 13 seq_num 108) dirty exists 2024-04-26T01:08:38.356 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:38.356 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:38.356 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:08:38.356 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:08:38.357 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:38.371 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:08:38.393 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:38.393+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:38.494 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:38.489+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:38.497 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:38.497 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:08:38.497 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:08:38.497 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:08:38.497 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.497 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 36/30 ECSubWrite(tid=15, reqid=client.4521.0:233, at_version=36'11, trim_to=0'0, roll_forward_to=33'9))", 2024-04-26T01:08:38.498 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:02.480147+0000", 2024-04-26T01:08:38.498 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 36.015569878999997, 2024-04-26T01:08:38.498 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.64949222799999995, 2024-04-26T01:08:38.498 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.498 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.498 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.498 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.498 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.498 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.498 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480147+0000", 2024-04-26T01:08:38.498 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.498 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.498 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.499 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480147+0000", 2024-04-26T01:08:38.499 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.499 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.499 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.499 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.499 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480148+0000", 2024-04-26T01:08:38.499 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2300000000000001e-06 2024-04-26T01:08:38.499 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.499 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.499 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.499 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480517+0000", 2024-04-26T01:08:38.499 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00036900700000000003 2024-04-26T01:08:38.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.500 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.500 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.500 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125262+0000", 2024-04-26T01:08:38.500 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.64474460600000005 2024-04-26T01:08:38.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.500 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.500 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.500 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125278+0000", 2024-04-26T01:08:38.500 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5622999999999999e-05 2024-04-26T01:08:38.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.500 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.500 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.500 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125483+0000", 2024-04-26T01:08:38.501 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000205105 2024-04-26T01:08:38.501 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.501 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.501 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.501 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125502+0000", 2024-04-26T01:08:38.501 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8876999999999999e-05 2024-04-26T01:08:38.501 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.501 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.501 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.501 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.129565+0000", 2024-04-26T01:08:38.501 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0040633930000000002 2024-04-26T01:08:38.501 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.501 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.501 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.502 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.129639+0000", 2024-04-26T01:08:38.502 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4387000000000005e-05 2024-04-26T01:08:38.502 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.502 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.502 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.502 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.502 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.502 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.5s6 43/30 ECSubWrite(tid=35, reqid=osd.15.0:35, at_version=43'30, trim_to=0'0, roll_forward_to=42'28))", 2024-04-26T01:08:38.502 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:38.502 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 29.089158138999998, 2024-04-26T01:08:38.502 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011310863000000001, 2024-04-26T01:08:38.502 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.502 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.503 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.503 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.503 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.503 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.503 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:38.503 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.503 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.503 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.503 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.503 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:38.503 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.503 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.503 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.503 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.504 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:38.504 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2e-07 2024-04-26T01:08:38.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.504 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.504 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.504 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406612+0000", 2024-04-26T01:08:38.504 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2206000000000001e-05 2024-04-26T01:08:38.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.504 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.504 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.504 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406709+0000", 2024-04-26T01:08:38.504 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7106000000000002e-05 2024-04-26T01:08:38.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.504 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.505 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.505 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406737+0000", 2024-04-26T01:08:38.505 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8629999999999999e-05 2024-04-26T01:08:38.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.505 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.505 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.505 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.407351+0000", 2024-04-26T01:08:38.505 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00061410800000000004 2024-04-26T01:08:38.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.505 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.505 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.505 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.407377+0000", 2024-04-26T01:08:38.505 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5057000000000001e-05 2024-04-26T01:08:38.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.506 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.506 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.506 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.417771+0000", 2024-04-26T01:08:38.506 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.01039402 2024-04-26T01:08:38.506 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.506 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.506 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.506 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.417870+0000", 2024-04-26T01:08:38.506 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9215999999999993e-05 2024-04-26T01:08:38.506 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.506 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.506 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.506 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.506 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.507 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 50/44 ECSubWrite(tid=51, reqid=osd.2.0:51, at_version=50'40, trim_to=0'0, roll_forward_to=45'38))", 2024-04-26T01:08:38.507 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.058137+0000", 2024-04-26T01:08:38.507 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 22.437579757000002, 2024-04-26T01:08:38.507 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0092019839999999999, 2024-04-26T01:08:38.507 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.507 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.507 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.507 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.507 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.507 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.507 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058137+0000", 2024-04-26T01:08:38.507 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.507 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.508 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.508 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058137+0000", 2024-04-26T01:08:38.508 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.508 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.508 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.508 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058138+0000", 2024-04-26T01:08:38.508 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.3300000000000001e-07 2024-04-26T01:08:38.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.508 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.508 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.508 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058197+0000", 2024-04-26T01:08:38.508 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9110000000000002e-05 2024-04-26T01:08:38.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.509 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.509 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.509 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058307+0000", 2024-04-26T01:08:38.509 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000109706 2024-04-26T01:08:38.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.509 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.509 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.509 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058330+0000", 2024-04-26T01:08:38.509 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3540000000000002e-05 2024-04-26T01:08:38.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.509 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.509 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.509 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058521+0000", 2024-04-26T01:08:38.509 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019083500000000001 2024-04-26T01:08:38.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.510 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.510 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.510 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058548+0000", 2024-04-26T01:08:38.510 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6950000000000001e-05 2024-04-26T01:08:38.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.510 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.510 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.510 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.067237+0000", 2024-04-26T01:08:38.510 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0086888769999999994 2024-04-26T01:08:38.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.510 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.510 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.511 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.067339+0000", 2024-04-26T01:08:38.511 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010213300000000001 2024-04-26T01:08:38.511 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.511 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.511 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.511 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.511 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.511 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs4 50/44 ECSubWrite(tid=22, reqid=osd.9.0:22, at_version=50'37, trim_to=0'0, roll_forward_to=47'35))", 2024-04-26T01:08:38.511 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:08:38.511 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 22.436508438000001, 2024-04-26T01:08:38.511 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016916468, 2024-04-26T01:08:38.511 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.511 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.512 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.512 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.512 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.512 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.512 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:08:38.512 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.512 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.512 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.512 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.512 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:08:38.512 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.512 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.513 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.513 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.513 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:08:38.513 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9299999999999996e-07 2024-04-26T01:08:38.513 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.513 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.513 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.513 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059282+0000", 2024-04-26T01:08:38.513 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2019000000000003e-05 2024-04-26T01:08:38.513 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.513 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.513 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.513 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059397+0000", 2024-04-26T01:08:38.513 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000115569 2024-04-26T01:08:38.514 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.514 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.514 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.514 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059419+0000", 2024-04-26T01:08:38.514 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1497000000000001e-05 2024-04-26T01:08:38.514 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.514 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.514 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.514 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064015+0000", 2024-04-26T01:08:38.514 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0045966059999999996 2024-04-26T01:08:38.514 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.515 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.515 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.515 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064041+0000", 2024-04-26T01:08:38.515 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5570000000000001e-05 2024-04-26T01:08:38.515 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.515 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.515 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.515 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.075972+0000", 2024-04-26T01:08:38.515 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011930802000000001 2024-04-26T01:08:38.515 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.515 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.515 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076125+0000", 2024-04-26T01:08:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015351199999999999 2024-04-26T01:08:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 50/44 ECSubWrite(tid=215, reqid=client.4521.0:490, at_version=50'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-26T01:08:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.059258+0000", 2024-04-26T01:08:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 22.436458565999999, 2024-04-26T01:08:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014073085000000001, 2024-04-26T01:08:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059258+0000", 2024-04-26T01:08:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059258+0000", 2024-04-26T01:08:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059261+0000", 2024-04-26T01:08:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3829999999999999e-06 2024-04-26T01:08:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059926+0000", 2024-04-26T01:08:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00066483199999999997 2024-04-26T01:08:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060011+0000", 2024-04-26T01:08:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5248999999999996e-05 2024-04-26T01:08:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060033+0000", 2024-04-26T01:08:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1566999999999999e-05 2024-04-26T01:08:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060387+0000", 2024-04-26T01:08:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000354793 2024-04-26T01:08:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060449+0000", 2024-04-26T01:08:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.1957000000000003e-05 2024-04-26T01:08:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073174+0000", 2024-04-26T01:08:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012724998 2024-04-26T01:08:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073332+0000", 2024-04-26T01:08:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015730599999999999 2024-04-26T01:08:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.0s5 50/30 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=50'32, trim_to=0'0, roll_forward_to=45'30))", 2024-04-26T01:08:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.062205+0000", 2024-04-26T01:08:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 22.433512186000002, 2024-04-26T01:08:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.01120789, 2024-04-26T01:08:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062205+0000", 2024-04-26T01:08:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062205+0000", 2024-04-26T01:08:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062207+0000", 2024-04-26T01:08:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.844e-06 2024-04-26T01:08:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062258+0000", 2024-04-26T01:08:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.1388999999999999e-05 2024-04-26T01:08:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.525 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.525 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062376+0000", 2024-04-26T01:08:38.525 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00011767199999999999 2024-04-26T01:08:38.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.525 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.525 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.525 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062399+0000", 2024-04-26T01:08:38.525 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3410000000000001e-05 2024-04-26T01:08:38.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.526 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.526 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.526 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062583+0000", 2024-04-26T01:08:38.526 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00018336900000000001 2024-04-26T01:08:38.526 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.526 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.526 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.526 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062611+0000", 2024-04-26T01:08:38.526 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8189999999999999e-05 2024-04-26T01:08:38.526 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.526 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.527 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.527 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073316+0000", 2024-04-26T01:08:38.527 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010704824 2024-04-26T01:08:38.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.527 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.527 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.527 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073413+0000", 2024-04-26T01:08:38.527 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7192000000000001e-05 2024-04-26T01:08:38.527 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.527 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.527 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.527 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.528 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.2s3 50/44 ECSubWrite(tid=212, reqid=osd.13.0:212, at_version=50'24, trim_to=0'0, roll_forward_to=45'22))", 2024-04-26T01:08:38.528 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.063661+0000", 2024-04-26T01:08:38.528 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 22.432055567999999, 2024-04-26T01:08:38.528 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012570563999999999, 2024-04-26T01:08:38.528 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.528 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.528 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.528 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.528 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.528 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.528 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063661+0000", 2024-04-26T01:08:38.528 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.528 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.528 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.529 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.529 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063661+0000", 2024-04-26T01:08:38.529 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.529 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.529 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063662+0000", 2024-04-26T01:08:38.529 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.37e-07 2024-04-26T01:08:38.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.529 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.529 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063721+0000", 2024-04-26T01:08:38.529 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.8696999999999997e-05 2024-04-26T01:08:38.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.530 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.530 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063860+0000", 2024-04-26T01:08:38.530 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013883799999999999 2024-04-26T01:08:38.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.530 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.530 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063882+0000", 2024-04-26T01:08:38.530 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2313000000000001e-05 2024-04-26T01:08:38.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.530 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.530 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064661+0000", 2024-04-26T01:08:38.531 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00077955699999999995 2024-04-26T01:08:38.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.531 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.531 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064686+0000", 2024-04-26T01:08:38.531 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4984e-05 2024-04-26T01:08:38.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.531 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.531 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.075967+0000", 2024-04-26T01:08:38.531 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011280646 2024-04-26T01:08:38.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.531 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076232+0000", 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00026499200000000003 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 50/30 ECSubWrite(tid=216, reqid=osd.4.0:214, at_version=0'0, trim_to=0'0, roll_forward_to=50'22))", 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.065107+0000", 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 22.430609665999999, 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0087328609999999997, 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.533 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.533 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065107+0000", 2024-04-26T01:08:38.533 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.533 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.533 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065107+0000", 2024-04-26T01:08:38.533 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.533 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.533 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065108+0000", 2024-04-26T01:08:38.533 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.0999999999999997e-07 2024-04-26T01:08:38.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.534 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.534 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065125+0000", 2024-04-26T01:08:38.534 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7010000000000001e-05 2024-04-26T01:08:38.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.534 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.534 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.534 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065360+0000", 2024-04-26T01:08:38.534 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00023507199999999999 2024-04-26T01:08:38.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.534 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.535 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.535 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065382+0000", 2024-04-26T01:08:38.535 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1993000000000001e-05 2024-04-26T01:08:38.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.535 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.535 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.535 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065672+0000", 2024-04-26T01:08:38.535 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00028928100000000001 2024-04-26T01:08:38.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.535 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.535 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.535 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065696+0000", 2024-04-26T01:08:38.535 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4273e-05 2024-04-26T01:08:38.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.536 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.536 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.536 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073748+0000", 2024-04-26T01:08:38.536 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0080520490000000004 2024-04-26T01:08:38.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.536 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.536 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.536 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073840+0000", 2024-04-26T01:08:38.536 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2373000000000005e-05 2024-04-26T01:08:38.536 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.536 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.536 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.536 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.537 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 50/44 ECSubWrite(tid=218, reqid=osd.4.0:218, at_version=50'34, trim_to=0'0, roll_forward_to=47'30))", 2024-04-26T01:08:38.537 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.065939+0000", 2024-04-26T01:08:38.537 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 22.429778052, 2024-04-26T01:08:38.537 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010065406000000001, 2024-04-26T01:08:38.537 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.537 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.537 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.537 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.537 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.537 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.537 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065939+0000", 2024-04-26T01:08:38.537 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.538 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.538 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065939+0000", 2024-04-26T01:08:38.538 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.538 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.538 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065941+0000", 2024-04-26T01:08:38.538 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9199999999999998e-06 2024-04-26T01:08:38.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.538 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.539 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065994+0000", 2024-04-26T01:08:38.539 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3439000000000001e-05 2024-04-26T01:08:38.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.539 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.539 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.539 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066088+0000", 2024-04-26T01:08:38.539 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.365e-05 2024-04-26T01:08:38.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.539 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.539 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.539 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066110+0000", 2024-04-26T01:08:38.539 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1912999999999998e-05 2024-04-26T01:08:38.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.539 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.540 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.540 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066779+0000", 2024-04-26T01:08:38.540 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00066918399999999997 2024-04-26T01:08:38.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.540 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.540 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.540 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066810+0000", 2024-04-26T01:08:38.540 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0446999999999998e-05 2024-04-26T01:08:38.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.540 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.540 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.540 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.075902+0000", 2024-04-26T01:08:38.540 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0090923529999999992 2024-04-26T01:08:38.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.541 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.541 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076004+0000", 2024-04-26T01:08:38.541 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0001025 2024-04-26T01:08:38.541 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.541 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.541 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.541 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.8s6 50/30 ECSubWrite(tid=42, reqid=osd.5.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=50'26))", 2024-04-26T01:08:38.541 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.068230+0000", 2024-04-26T01:08:38.541 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 22.427487201000002, 2024-04-26T01:08:38.541 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0080202260000000001, 2024-04-26T01:08:38.541 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.542 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.542 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.542 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.542 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.542 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068230+0000", 2024-04-26T01:08:38.542 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.542 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.542 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068230+0000", 2024-04-26T01:08:38.542 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.543 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.543 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068232+0000", 2024-04-26T01:08:38.543 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.804e-06 2024-04-26T01:08:38.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.543 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.543 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.543 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068261+0000", 2024-04-26T01:08:38.543 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9595999999999999e-05 2024-04-26T01:08:38.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.543 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.543 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.543 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068341+0000", 2024-04-26T01:08:38.543 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9302999999999996e-05 2024-04-26T01:08:38.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.544 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.544 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.544 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068355+0000", 2024-04-26T01:08:38.544 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.465e-05 2024-04-26T01:08:38.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.544 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.544 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.544 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068495+0000", 2024-04-26T01:08:38.544 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013948500000000001 2024-04-26T01:08:38.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.544 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.544 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.544 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068520+0000", 2024-04-26T01:08:38.544 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5015999999999999e-05 2024-04-26T01:08:38.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.545 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.545 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.545 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076177+0000", 2024-04-26T01:08:38.545 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0076572489999999997 2024-04-26T01:08:38.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.545 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.545 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.545 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076250+0000", 2024-04-26T01:08:38.545 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.3122999999999998e-05 2024-04-26T01:08:38.545 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.545 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.545 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.546 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.546 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 55/44 ECSubWrite(tid=253, reqid=osd.4.0:253, at_version=55'43, trim_to=0'0, roll_forward_to=52'41))", 2024-04-26T01:08:38.546 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:21.084283+0000", 2024-04-26T01:08:38.546 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 17.411434369999998, 2024-04-26T01:08:38.546 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061481820000000003, 2024-04-26T01:08:38.546 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.546 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.546 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.546 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.546 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.546 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.546 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084283+0000", 2024-04-26T01:08:38.546 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.547 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.547 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.547 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084283+0000", 2024-04-26T01:08:38.547 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.547 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.547 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.547 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084283+0000", 2024-04-26T01:08:38.547 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0699999999999997e-07 2024-04-26T01:08:38.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.548 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.548 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.548 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084362+0000", 2024-04-26T01:08:38.548 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9083000000000002e-05 2024-04-26T01:08:38.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.548 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.548 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.548 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084566+0000", 2024-04-26T01:08:38.548 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00020390800000000001 2024-04-26T01:08:38.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.548 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.548 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.548 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084589+0000", 2024-04-26T01:08:38.549 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3213e-05 2024-04-26T01:08:38.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.549 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.549 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.549 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084751+0000", 2024-04-26T01:08:38.549 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016103600000000001 2024-04-26T01:08:38.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.549 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.549 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.549 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084781+0000", 2024-04-26T01:08:38.550 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0372999999999999e-05 2024-04-26T01:08:38.550 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.550 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.550 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.550 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.090337+0000", 2024-04-26T01:08:38.550 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.005556503 2024-04-26T01:08:38.550 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.550 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.550 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.550 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.090431+0000", 2024-04-26T01:08:38.550 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.3459000000000001e-05 2024-04-26T01:08:38.550 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.551 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.551 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.551 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.551 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 55/30 ECSubWrite(tid=252, reqid=client.4521.0:548, at_version=55'28, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:38.551 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:21.084368+0000", 2024-04-26T01:08:38.551 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 17.411348802999999, 2024-04-26T01:08:38.551 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0068931720000000004, 2024-04-26T01:08:38.551 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.551 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.552 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.552 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.552 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.552 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.552 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084368+0000", 2024-04-26T01:08:38.552 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.552 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.552 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.552 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.552 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084368+0000", 2024-04-26T01:08:38.552 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.552 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.552 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.553 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.553 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084369+0000", 2024-04-26T01:08:38.553 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.1999999999999999e-07 2024-04-26T01:08:38.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.553 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.553 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.553 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085034+0000", 2024-04-26T01:08:38.553 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00066459099999999997 2024-04-26T01:08:38.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.553 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.553 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.554 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085069+0000", 2024-04-26T01:08:38.554 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4965999999999997e-05 2024-04-26T01:08:38.554 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.554 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.554 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.554 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085081+0000", 2024-04-26T01:08:38.554 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2422999999999999e-05 2024-04-26T01:08:38.554 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.554 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.554 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.554 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085243+0000", 2024-04-26T01:08:38.555 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000162466 2024-04-26T01:08:38.555 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.555 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.555 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.555 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085270+0000", 2024-04-26T01:08:38.555 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6775999999999999e-05 2024-04-26T01:08:38.555 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.555 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.555 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.555 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091175+0000", 2024-04-26T01:08:38.555 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059045340000000003 2024-04-26T01:08:38.556 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.556 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.556 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.556 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091261+0000", 2024-04-26T01:08:38.556 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.6695999999999998e-05 2024-04-26T01:08:38.556 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.556 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.556 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.556 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.556 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.556 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 55/44 ECSubWrite(tid=55, reqid=osd.2.0:55, at_version=55'44, trim_to=0'0, roll_forward_to=52'42))", 2024-04-26T01:08:38.557 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:21.093675+0000", 2024-04-26T01:08:38.557 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 17.402041886999999, 2024-04-26T01:08:38.557 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066519099999999996, 2024-04-26T01:08:38.557 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.557 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.557 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.557 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.557 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.557 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.557 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093675+0000", 2024-04-26T01:08:38.558 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.558 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.558 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093675+0000", 2024-04-26T01:08:38.558 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.558 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.559 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093676+0000", 2024-04-26T01:08:38.559 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.3e-07 2024-04-26T01:08:38.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.559 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.559 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.559 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093734+0000", 2024-04-26T01:08:38.559 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7833e-05 2024-04-26T01:08:38.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.559 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.559 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.559 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093836+0000", 2024-04-26T01:08:38.559 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010234900000000001 2024-04-26T01:08:38.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.560 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.560 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.560 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093859+0000", 2024-04-26T01:08:38.560 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3383e-05 2024-04-26T01:08:38.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.560 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.560 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.560 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.094252+0000", 2024-04-26T01:08:38.560 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000392424 2024-04-26T01:08:38.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.560 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.560 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.561 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.094287+0000", 2024-04-26T01:08:38.561 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5713000000000002e-05 2024-04-26T01:08:38.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.561 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.561 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.561 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.100231+0000", 2024-04-26T01:08:38.561 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059430519999999999 2024-04-26T01:08:38.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.561 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.561 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.561 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.100327+0000", 2024-04-26T01:08:38.561 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6526000000000004e-05 2024-04-26T01:08:38.561 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.562 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.562 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.562 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.562 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 59/30 ECSubWrite(tid=320, reqid=osd.4.0:320, at_version=59'43, trim_to=0'0, roll_forward_to=58'41))", 2024-04-26T01:08:38.562 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.159033+0000", 2024-04-26T01:08:38.562 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.336684194, 2024-04-26T01:08:38.562 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0060271350000000003, 2024-04-26T01:08:38.562 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.562 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.562 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.562 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.562 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.563 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.563 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159033+0000", 2024-04-26T01:08:38.563 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.563 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.563 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159033+0000", 2024-04-26T01:08:38.563 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.563 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.563 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159033+0000", 2024-04-26T01:08:38.563 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9699999999999996e-07 2024-04-26T01:08:38.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159076+0000", 2024-04-26T01:08:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.2616e-05 2024-04-26T01:08:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159156+0000", 2024-04-26T01:08:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.0400000000000003e-05 2024-04-26T01:08:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159180+0000", 2024-04-26T01:08:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3898999999999999e-05 2024-04-26T01:08:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159352+0000", 2024-04-26T01:08:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017211599999999999 2024-04-26T01:08:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.159382+0000", 2024-04-26T01:08:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9410000000000001e-05 2024-04-26T01:08:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.164958+0000", 2024-04-26T01:08:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.005575628 2024-04-26T01:08:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.165060+0000", 2024-04-26T01:08:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000102469 2024-04-26T01:08:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 59/30 ECSubWrite(tid=322, reqid=osd.4.0:322, at_version=59'45, trim_to=0'0, roll_forward_to=58'41))", 2024-04-26T01:08:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.160873+0000", 2024-04-26T01:08:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.334844361, 2024-04-26T01:08:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0064776979999999996, 2024-04-26T01:08:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160873+0000", 2024-04-26T01:08:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160873+0000", 2024-04-26T01:08:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160874+0000", 2024-04-26T01:08:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5030000000000001e-06 2024-04-26T01:08:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160924+0000", 2024-04-26T01:08:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0158999999999999e-05 2024-04-26T01:08:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161010+0000", 2024-04-26T01:08:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5285999999999996e-05 2024-04-26T01:08:38.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.569 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.569 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161032+0000", 2024-04-26T01:08:38.569 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2427000000000001e-05 2024-04-26T01:08:38.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.569 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.569 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161983+0000", 2024-04-26T01:08:38.569 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00095064900000000003 2024-04-26T01:08:38.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.569 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.569 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.162010+0000", 2024-04-26T01:08:38.569 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7786000000000001e-05 2024-04-26T01:08:38.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.570 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.570 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167263+0000", 2024-04-26T01:08:38.570 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0052521979999999996 2024-04-26T01:08:38.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.570 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.570 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167350+0000", 2024-04-26T01:08:38.570 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7689999999999996e-05 2024-04-26T01:08:38.570 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.570 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.570 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.571 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs4 60/44 ECSubWrite(tid=50, reqid=osd.9.0:50, at_version=60'60, trim_to=0'0, roll_forward_to=59'56))", 2024-04-26T01:08:38.571 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.190277+0000", 2024-04-26T01:08:38.571 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 12.305439998000001, 2024-04-26T01:08:38.571 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0056685950000000002, 2024-04-26T01:08:38.571 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.571 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.571 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.571 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.571 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.571 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190277+0000", 2024-04-26T01:08:38.571 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.572 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.572 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190277+0000", 2024-04-26T01:08:38.572 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.572 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.572 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190279+0000", 2024-04-26T01:08:38.572 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9300000000000002e-06 2024-04-26T01:08:38.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.572 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.572 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190330+0000", 2024-04-26T01:08:38.572 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0798999999999999e-05 2024-04-26T01:08:38.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.573 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.573 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190406+0000", 2024-04-26T01:08:38.573 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.6406e-05 2024-04-26T01:08:38.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.573 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.573 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190427+0000", 2024-04-26T01:08:38.573 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0590000000000001e-05 2024-04-26T01:08:38.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.573 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.573 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190589+0000", 2024-04-26T01:08:38.574 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000161872 2024-04-26T01:08:38.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.574 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.574 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.190626+0000", 2024-04-26T01:08:38.574 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7403000000000001e-05 2024-04-26T01:08:38.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.574 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.574 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195858+0000", 2024-04-26T01:08:38.574 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0052319289999999997 2024-04-26T01:08:38.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.575 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.575 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.195946+0000", 2024-04-26T01:08:38.575 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7665999999999997e-05 2024-04-26T01:08:38.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.575 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.575 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.0s5 60/30 ECSubWrite(tid=71, reqid=osd.14.0:70, at_version=0'0, trim_to=0'0, roll_forward_to=60'57))", 2024-04-26T01:08:38.575 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.214164+0000", 2024-04-26T01:08:38.575 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 12.281553168, 2024-04-26T01:08:38.575 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059055820000000004, 2024-04-26T01:08:38.575 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.576 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.576 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.576 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.576 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.576 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214164+0000", 2024-04-26T01:08:38.576 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.576 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.576 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214164+0000", 2024-04-26T01:08:38.576 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.577 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.577 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214166+0000", 2024-04-26T01:08:38.577 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1660000000000001e-06 2024-04-26T01:08:38.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.577 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.577 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214194+0000", 2024-04-26T01:08:38.577 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8067000000000001e-05 2024-04-26T01:08:38.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.577 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.577 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214283+0000", 2024-04-26T01:08:38.577 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8609000000000005e-05 2024-04-26T01:08:38.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.578 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.578 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214306+0000", 2024-04-26T01:08:38.578 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3087000000000001e-05 2024-04-26T01:08:38.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.578 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.578 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214457+0000", 2024-04-26T01:08:38.578 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015159799999999999 2024-04-26T01:08:38.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.578 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.578 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.214491+0000", 2024-04-26T01:08:38.579 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.3166000000000001e-05 2024-04-26T01:08:38.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.579 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.579 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.579 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.219992+0000", 2024-04-26T01:08:38.579 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0055015009999999998 2024-04-26T01:08:38.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.579 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.579 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.579 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.220069+0000", 2024-04-26T01:08:38.579 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.7387999999999998e-05 2024-04-26T01:08:38.579 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.579 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.580 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.580 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.580 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 64/44 ECSubWrite(tid=388, reqid=client.4521.0:693, at_version=64'70, trim_to=0'0, roll_forward_to=61'68))", 2024-04-26T01:08:38.580 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:30.208877+0000", 2024-04-26T01:08:38.580 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 8.2868404669999993, 2024-04-26T01:08:38.580 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0057091679999999997, 2024-04-26T01:08:38.580 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.580 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.580 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.581 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.581 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.581 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.581 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.208877+0000", 2024-04-26T01:08:38.581 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.581 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.581 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.581 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.208877+0000", 2024-04-26T01:08:38.581 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.581 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.581 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.581 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.208877+0000", 2024-04-26T01:08:38.582 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.6000000000000002e-07 2024-04-26T01:08:38.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.582 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.582 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.582 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.210397+0000", 2024-04-26T01:08:38.582 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0015195779999999999 2024-04-26T01:08:38.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.582 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.582 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.582 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.210495+0000", 2024-04-26T01:08:38.583 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.8465e-05 2024-04-26T01:08:38.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.583 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.583 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.210520+0000", 2024-04-26T01:08:38.583 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.423e-05 2024-04-26T01:08:38.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.583 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.583 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.210690+0000", 2024-04-26T01:08:38.583 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016986200000000001 2024-04-26T01:08:38.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.584 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.584 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.210720+0000", 2024-04-26T01:08:38.584 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0880000000000002e-05 2024-04-26T01:08:38.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.584 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.584 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.214492+0000", 2024-04-26T01:08:38.584 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0037718600000000001 2024-04-26T01:08:38.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.584 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.584 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.214586+0000", 2024-04-26T01:08:38.584 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.3432999999999998e-05 2024-04-26T01:08:38.584 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.584 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.585 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.585 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs4 70/44 ECSubWrite(tid=70, reqid=osd.9.0:70, at_version=70'78, trim_to=0'0, roll_forward_to=69'76))", 2024-04-26T01:08:38.585 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:36.069042+0000", 2024-04-26T01:08:38.585 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 2.4266748900000001, 2024-04-26T01:08:38.585 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0063037789999999998, 2024-04-26T01:08:38.585 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.585 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.585 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.585 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.585 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.585 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069042+0000", 2024-04-26T01:08:38.586 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.586 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.586 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.586 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069042+0000", 2024-04-26T01:08:38.586 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.586 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.586 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.586 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069043+0000", 2024-04-26T01:08:38.586 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0699999999999997e-07 2024-04-26T01:08:38.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.586 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.586 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.587 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069117+0000", 2024-04-26T01:08:38.587 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4338999999999994e-05 2024-04-26T01:08:38.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.587 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.587 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069244+0000", 2024-04-26T01:08:38.587 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000127222 2024-04-26T01:08:38.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.587 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.587 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069270+0000", 2024-04-26T01:08:38.587 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5633e-05 2024-04-26T01:08:38.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.588 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.588 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.588 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069475+0000", 2024-04-26T01:08:38.588 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00020550200000000001 2024-04-26T01:08:38.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.588 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.588 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.588 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069507+0000", 2024-04-26T01:08:38.588 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1883e-05 2024-04-26T01:08:38.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.588 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.588 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.589 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.075253+0000", 2024-04-26T01:08:38.589 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0057461539999999998 2024-04-26T01:08:38.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.589 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.589 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.589 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.075346+0000", 2024-04-26T01:08:38.589 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2539000000000003e-05 2024-04-26T01:08:38.589 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.589 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.589 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.589 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.590 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.5s6 72/30 ECSubWrite(tid=56, reqid=client.4521.0:785, at_version=72'49, trim_to=0'0, roll_forward_to=72'47))", 2024-04-26T01:08:38.590 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:38.301857+0000", 2024-04-26T01:08:38.590 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.19386029199999999, 2024-04-26T01:08:38.590 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0073268409999999997, 2024-04-26T01:08:38.590 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:38.590 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:38.590 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:38.590 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:38.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.590 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.301857+0000", 2024-04-26T01:08:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.301857+0000", 2024-04-26T01:08:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.301859+0000", 2024-04-26T01:08:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.588e-06 2024-04-26T01:08:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.306215+0000", 2024-04-26T01:08:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0043557179999999997 2024-04-26T01:08:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.306269+0000", 2024-04-26T01:08:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.4289999999999997e-05 2024-04-26T01:08:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.306282+0000", 2024-04-26T01:08:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2332e-05 2024-04-26T01:08:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.306357+0000", 2024-04-26T01:08:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5117999999999995e-05 2024-04-26T01:08:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.306368+0000", 2024-04-26T01:08:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1063000000000001e-05 2024-04-26T01:08:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.309146+0000", 2024-04-26T01:08:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0027777040000000002 2024-04-26T01:08:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.309184+0000", 2024-04-26T01:08:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8028000000000003e-05 2024-04-26T01:08:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:38.596 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:38.596 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:38.597 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:08:38.598 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:38.592+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:38.699 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:38.693+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:38.724 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:38.724 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:38.724 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:08:38.724 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:38.739 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:08:38.799 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:38.795+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:38.812 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:08:38.854 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:38.855 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:38.855 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:08:38.855 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:08:38.855 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:38.869 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:08:38.901 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:38.899+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:38.982 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:38.982 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:39.001 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:38.999+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:39.009 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:39.009 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:39.037 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:08:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:08:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:08:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 32/30 ECSubWrite(tid=1, reqid=client.4521.0:25, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 39.721164549999997, 2024-04-26T01:08:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062762640000000001, 2024-04-26T01:08:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2700000000000001e-07 2024-04-26T01:08:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313923+0000", 2024-04-26T01:08:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.0530000000000002e-05 2024-04-26T01:08:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314064+0000", 2024-04-26T01:08:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000140841 2024-04-26T01:08:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314070+0000", 2024-04-26T01:08:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.976e-06 2024-04-26T01:08:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314140+0000", 2024-04-26T01:08:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9664000000000003e-05 2024-04-26T01:08:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314151+0000", 2024-04-26T01:08:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1251e-05 2024-04-26T01:08:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320065+0000", 2024-04-26T01:08:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059141990000000002 2024-04-26T01:08:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320109+0000", 2024-04-26T01:08:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3575999999999997e-05 2024-04-26T01:08:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.ds3 33/30 ECSubWrite(tid=5, reqid=client.4521.0:35, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.325729+0000", 2024-04-26T01:08:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 39.70926789, 2024-04-26T01:08:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079681170000000003, 2024-04-26T01:08:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325729+0000", 2024-04-26T01:08:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325729+0000", 2024-04-26T01:08:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325730+0000", 2024-04-26T01:08:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.78e-07 2024-04-26T01:08:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325748+0000", 2024-04-26T01:08:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8127000000000001e-05 2024-04-26T01:08:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325789+0000", 2024-04-26T01:08:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.1100000000000003e-05 2024-04-26T01:08:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325800+0000", 2024-04-26T01:08:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0256e-05 2024-04-26T01:08:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325859+0000", 2024-04-26T01:08:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9849999999999998e-05 2024-04-26T01:08:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325869+0000", 2024-04-26T01:08:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6989999999999998e-06 2024-04-26T01:08:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.333645+0000", 2024-04-26T01:08:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0077756800000000001 2024-04-26T01:08:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.333697+0000", 2024-04-26T01:08:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2426999999999998e-05 2024-04-26T01:08:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 33/30 ECSubWrite(tid=4, reqid=client.4521.0:26, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.327582+0000", 2024-04-26T01:08:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 39.707415249999997, 2024-04-26T01:08:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0074218790000000001, 2024-04-26T01:08:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327582+0000", 2024-04-26T01:08:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327582+0000", 2024-04-26T01:08:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327583+0000", 2024-04-26T01:08:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0669999999999999e-06 2024-04-26T01:08:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327596+0000", 2024-04-26T01:08:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3135e-05 2024-04-26T01:08:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327649+0000", 2024-04-26T01:08:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2704999999999998e-05 2024-04-26T01:08:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327659+0000", 2024-04-26T01:08:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.8470000000000008e-06 2024-04-26T01:08:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327733+0000", 2024-04-26T01:08:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4114000000000002e-05 2024-04-26T01:08:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327746+0000", 2024-04-26T01:08:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3176999999999999e-05 2024-04-26T01:08:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.334969+0000", 2024-04-26T01:08:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0072232160000000002 2024-04-26T01:08:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.335004+0000", 2024-04-26T01:08:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4618e-05 2024-04-26T01:08:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.053 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 50/44 ECSubWrite(tid=213, reqid=osd.4.0:213, at_version=50'36, trim_to=0'0, roll_forward_to=47'34))", 2024-04-26T01:08:39.053 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.047619+0000", 2024-04-26T01:08:39.053 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 22.987378069000002, 2024-04-26T01:08:39.053 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012971391000000001, 2024-04-26T01:08:39.053 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.053 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.053 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.053 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.053 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047619+0000", 2024-04-26T01:08:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047619+0000", 2024-04-26T01:08:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047620+0000", 2024-04-26T01:08:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9999999999999997e-07 2024-04-26T01:08:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047663+0000", 2024-04-26T01:08:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3254000000000003e-05 2024-04-26T01:08:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047762+0000", 2024-04-26T01:08:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9551999999999998e-05 2024-04-26T01:08:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047793+0000", 2024-04-26T01:08:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0729999999999999e-05 2024-04-26T01:08:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.048521+0000", 2024-04-26T01:08:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000727971 2024-04-26T01:08:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.048552+0000", 2024-04-26T01:08:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1375999999999999e-05 2024-04-26T01:08:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060503+0000", 2024-04-26T01:08:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011950771000000001 2024-04-26T01:08:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060590+0000", 2024-04-26T01:08:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7137000000000003e-05 2024-04-26T01:08:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.059 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 50/44 ECSubWrite(tid=217, reqid=osd.4.0:213, at_version=0'0, trim_to=0'0, roll_forward_to=50'36))", 2024-04-26T01:08:39.059 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.063223+0000", 2024-04-26T01:08:39.059 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 22.971773593000002, 2024-04-26T01:08:39.059 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0071347260000000001, 2024-04-26T01:08:39.059 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.059 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.059 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.059 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.059 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063223+0000", 2024-04-26T01:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063223+0000", 2024-04-26T01:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063225+0000", 2024-04-26T01:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.753e-06 2024-04-26T01:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063249+0000", 2024-04-26T01:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3896999999999998e-05 2024-04-26T01:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063327+0000", 2024-04-26T01:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.7693000000000003e-05 2024-04-26T01:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063349+0000", 2024-04-26T01:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2446000000000002e-05 2024-04-26T01:08:39.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.062 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.062 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063511+0000", 2024-04-26T01:08:39.062 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016191799999999999 2024-04-26T01:08:39.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.062 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.062 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063538+0000", 2024-04-26T01:08:39.062 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7237e-05 2024-04-26T01:08:39.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.062 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.063 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.070265+0000", 2024-04-26T01:08:39.063 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.006726526 2024-04-26T01:08:39.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.063 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.063 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.070358+0000", 2024-04-26T01:08:39.063 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.3256000000000003e-05 2024-04-26T01:08:39.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.063 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.063 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.1s4 50/44 ECSubWrite(tid=219, reqid=osd.4.0:219, at_version=50'58, trim_to=0'0, roll_forward_to=47'56))", 2024-04-26T01:08:39.064 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.066105+0000", 2024-04-26T01:08:39.064 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 22.968892569000001, 2024-04-26T01:08:39.064 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079801060000000007, 2024-04-26T01:08:39.064 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.064 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.064 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.064 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.064 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.064 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.064 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066105+0000", 2024-04-26T01:08:39.064 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.064 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.064 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.064 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066105+0000", 2024-04-26T01:08:39.065 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.065 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.065 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.065 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066106+0000", 2024-04-26T01:08:39.065 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.846e-06 2024-04-26T01:08:39.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.065 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.065 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.065 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066156+0000", 2024-04-26T01:08:39.065 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9225999999999999e-05 2024-04-26T01:08:39.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.065 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.065 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.066 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066238+0000", 2024-04-26T01:08:39.066 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2126999999999995e-05 2024-04-26T01:08:39.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.066 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.066 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.066 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066260+0000", 2024-04-26T01:08:39.066 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2408999999999998e-05 2024-04-26T01:08:39.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.066 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.066 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.066 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066417+0000", 2024-04-26T01:08:39.066 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000157312 2024-04-26T01:08:39.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.067 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.067 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.067 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066443+0000", 2024-04-26T01:08:39.067 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5797000000000001e-05 2024-04-26T01:08:39.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.067 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.067 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.067 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073990+0000", 2024-04-26T01:08:39.067 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0075467670000000002 2024-04-26T01:08:39.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.068 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.068 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.068 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.074085+0000", 2024-04-26T01:08:39.068 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.4622000000000003e-05 2024-04-26T01:08:39.068 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.068 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.068 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.068 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.068 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 50/44 ECSubWrite(tid=220, reqid=client.4521.0:491, at_version=50'38, trim_to=0'0, roll_forward_to=50'36))", 2024-04-26T01:08:39.068 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.072911+0000", 2024-04-26T01:08:39.068 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 22.962086047, 2024-04-26T01:08:39.068 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0076486990000000001, 2024-04-26T01:08:39.069 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.069 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.069 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.069 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.069 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.069 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.069 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.072911+0000", 2024-04-26T01:08:39.069 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.069 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.069 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.069 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.069 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.072911+0000", 2024-04-26T01:08:39.069 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.069 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.069 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.070 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.070 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.072914+0000", 2024-04-26T01:08:39.070 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5900000000000002e-06 2024-04-26T01:08:39.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.070 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.070 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.070 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073741+0000", 2024-04-26T01:08:39.070 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00082785699999999999 2024-04-26T01:08:39.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.070 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.070 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.070 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073888+0000", 2024-04-26T01:08:39.070 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014690499999999999 2024-04-26T01:08:39.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.071 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.071 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.071 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073915+0000", 2024-04-26T01:08:39.071 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.614e-05 2024-04-26T01:08:39.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.071 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.071 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.071 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.074074+0000", 2024-04-26T01:08:39.071 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015989500000000001 2024-04-26T01:08:39.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.071 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.071 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.071 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.074101+0000", 2024-04-26T01:08:39.071 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6693999999999999e-05 2024-04-26T01:08:39.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.072 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.072 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.072 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080467+0000", 2024-04-26T01:08:39.072 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0063662689999999999 2024-04-26T01:08:39.072 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.072 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.072 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.072 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080560+0000", 2024-04-26T01:08:39.072 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2349000000000006e-05 2024-04-26T01:08:39.072 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.072 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.072 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.072 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.073 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.073 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 50/44 ECSubWrite(tid=23, reqid=osd.9.0:22, at_version=0'0, trim_to=0'0, roll_forward_to=50'37))", 2024-04-26T01:08:39.073 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.079931+0000", 2024-04-26T01:08:39.073 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 22.955066533, 2024-04-26T01:08:39.073 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066276529999999998, 2024-04-26T01:08:39.073 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.073 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.073 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.073 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.073 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.073 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.073 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079931+0000", 2024-04-26T01:08:39.073 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.073 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.074 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.074 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079931+0000", 2024-04-26T01:08:39.074 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.074 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.074 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.074 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079933+0000", 2024-04-26T01:08:39.074 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6759999999999999e-06 2024-04-26T01:08:39.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.074 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.074 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.074 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079965+0000", 2024-04-26T01:08:39.074 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1927000000000001e-05 2024-04-26T01:08:39.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.075 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.075 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.075 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080073+0000", 2024-04-26T01:08:39.075 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107796 2024-04-26T01:08:39.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.075 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.075 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.075 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080098+0000", 2024-04-26T01:08:39.075 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4822999999999999e-05 2024-04-26T01:08:39.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.075 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.075 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.075 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080244+0000", 2024-04-26T01:08:39.075 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014625799999999999 2024-04-26T01:08:39.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.076 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.076 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080268+0000", 2024-04-26T01:08:39.076 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4128e-05 2024-04-26T01:08:39.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.076 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.076 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.086463+0000", 2024-04-26T01:08:39.076 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061951300000000001 2024-04-26T01:08:39.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.076 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.076 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.086558+0000", 2024-04-26T01:08:39.076 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.4914999999999995e-05 2024-04-26T01:08:39.076 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.077 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.077 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.077 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 50/44 ECSubWrite(tid=222, reqid=client.4521.0:492, at_version=50'39, trim_to=0'0, roll_forward_to=50'36))", 2024-04-26T01:08:39.077 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.080404+0000", 2024-04-26T01:08:39.077 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 22.954592859000002, 2024-04-26T01:08:39.077 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062500250000000002, 2024-04-26T01:08:39.077 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.077 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.077 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.077 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.077 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.077 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080404+0000", 2024-04-26T01:08:39.078 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.078 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.078 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.078 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080404+0000", 2024-04-26T01:08:39.078 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.078 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.078 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.078 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080406+0000", 2024-04-26T01:08:39.078 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0109999999999999e-06 2024-04-26T01:08:39.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.078 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.078 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.078 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080566+0000", 2024-04-26T01:08:39.079 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015976200000000001 2024-04-26T01:08:39.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.079 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.079 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080666+0000", 2024-04-26T01:08:39.079 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9698999999999995e-05 2024-04-26T01:08:39.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.079 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.079 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080693+0000", 2024-04-26T01:08:39.079 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6934e-05 2024-04-26T01:08:39.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.080 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.080 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080851+0000", 2024-04-26T01:08:39.080 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015818599999999999 2024-04-26T01:08:39.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.080 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.080 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.080 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080875+0000", 2024-04-26T01:08:39.080 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4032999999999998e-05 2024-04-26T01:08:39.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.080 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.086584+0000", 2024-04-26T01:08:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0057096109999999999 2024-04-26T01:08:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.086654+0000", 2024-04-26T01:08:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9789000000000006e-05 2024-04-26T01:08:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 53/44 ECSubWrite(tid=249, reqid=client.4521.0:539, at_version=53'46, trim_to=0'0, roll_forward_to=51'44))", 2024-04-26T01:08:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:19.133290+0000", 2024-04-26T01:08:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 19.901707015, 2024-04-26T01:08:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079087370000000008, 2024-04-26T01:08:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.133290+0000", 2024-04-26T01:08:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.133290+0000", 2024-04-26T01:08:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.133291+0000", 2024-04-26T01:08:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.4700000000000001e-07 2024-04-26T01:08:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.134718+0000", 2024-04-26T01:08:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001426921 2024-04-26T01:08:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.134818+0000", 2024-04-26T01:08:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000100203 2024-04-26T01:08:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.134841+0000", 2024-04-26T01:08:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2730000000000001e-05 2024-04-26T01:08:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.135022+0000", 2024-04-26T01:08:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00018121500000000001 2024-04-26T01:08:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.135051+0000", 2024-04-26T01:08:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8920000000000001e-05 2024-04-26T01:08:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.086 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.086 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.141103+0000", 2024-04-26T01:08:39.086 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0060520549999999998 2024-04-26T01:08:39.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.086 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.086 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.141199+0000", 2024-04-26T01:08:39.086 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6045999999999995e-05 2024-04-26T01:08:39.086 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.086 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.086 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.087 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 59/44 ECSubWrite(tid=323, reqid=osd.4.0:323, at_version=59'77, trim_to=0'0, roll_forward_to=58'73))", 2024-04-26T01:08:39.087 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.160765+0000", 2024-04-26T01:08:39.087 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.874231996000001, 2024-04-26T01:08:39.087 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066099469999999997, 2024-04-26T01:08:39.087 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.087 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.087 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.087 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.087 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.087 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160765+0000", 2024-04-26T01:08:39.088 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.088 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.088 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160765+0000", 2024-04-26T01:08:39.088 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.088 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.088 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160766+0000", 2024-04-26T01:08:39.089 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.1999999999999999e-07 2024-04-26T01:08:39.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.089 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.089 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.089 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160804+0000", 2024-04-26T01:08:39.089 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8068999999999998e-05 2024-04-26T01:08:39.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.089 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.089 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.089 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161001+0000", 2024-04-26T01:08:39.090 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000197159 2024-04-26T01:08:39.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.090 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.090 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.090 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161025+0000", 2024-04-26T01:08:39.090 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3768999999999999e-05 2024-04-26T01:08:39.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.090 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.090 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.090 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.162206+0000", 2024-04-26T01:08:39.090 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001181398 2024-04-26T01:08:39.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.091 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.091 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.162230+0000", 2024-04-26T01:08:39.091 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.419e-05 2024-04-26T01:08:39.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.091 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.091 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167200+0000", 2024-04-26T01:08:39.091 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0049697099999999996 2024-04-26T01:08:39.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.092 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.092 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167375+0000", 2024-04-26T01:08:39.092 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017493200000000001 2024-04-26T01:08:39.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.092 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.093 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 59/44 ECSubWrite(tid=36, reqid=osd.9.0:36, at_version=59'49, trim_to=0'0, roll_forward_to=57'47))", 2024-04-26T01:08:39.093 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.172944+0000", 2024-04-26T01:08:39.093 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.862052914, 2024-04-26T01:08:39.093 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0060582149999999996, 2024-04-26T01:08:39.093 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.093 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.093 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.093 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.093 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.093 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.093 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.172944+0000", 2024-04-26T01:08:39.093 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.093 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.093 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.172944+0000", 2024-04-26T01:08:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.172945+0000", 2024-04-26T01:08:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5000000000000002e-07 2024-04-26T01:08:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.173004+0000", 2024-04-26T01:08:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9209999999999997e-05 2024-04-26T01:08:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.095 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.173118+0000", 2024-04-26T01:08:39.095 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000114422 2024-04-26T01:08:39.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.095 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.095 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.095 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.173143+0000", 2024-04-26T01:08:39.095 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4193999999999999e-05 2024-04-26T01:08:39.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.095 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.095 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.095 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.173323+0000", 2024-04-26T01:08:39.095 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000180104 2024-04-26T01:08:39.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.095 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.096 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.096 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.173340+0000", 2024-04-26T01:08:39.096 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.719e-05 2024-04-26T01:08:39.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.096 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.096 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.096 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.178897+0000", 2024-04-26T01:08:39.096 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0055569130000000001 2024-04-26T01:08:39.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.096 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.096 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.096 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.179002+0000", 2024-04-26T01:08:39.096 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010553200000000001 2024-04-26T01:08:39.096 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.096 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.097 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.097 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.097 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 59/44 ECSubWrite(tid=325, reqid=osd.4.0:323, at_version=0'0, trim_to=0'0, roll_forward_to=59'77))", 2024-04-26T01:08:39.097 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.175746+0000", 2024-04-26T01:08:39.097 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.859251043, 2024-04-26T01:08:39.097 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0064646620000000004, 2024-04-26T01:08:39.097 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.097 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.097 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.097 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.097 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.097 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.097 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175746+0000", 2024-04-26T01:08:39.098 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.098 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.098 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.098 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175746+0000", 2024-04-26T01:08:39.098 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.098 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.098 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.098 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175748+0000", 2024-04-26T01:08:39.098 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0140000000000001e-06 2024-04-26T01:08:39.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.098 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.098 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.098 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175783+0000", 2024-04-26T01:08:39.099 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4946000000000001e-05 2024-04-26T01:08:39.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.099 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.099 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175872+0000", 2024-04-26T01:08:39.099 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8722999999999997e-05 2024-04-26T01:08:39.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.099 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.099 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175895+0000", 2024-04-26T01:08:39.099 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3649999999999999e-05 2024-04-26T01:08:39.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.099 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.099 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.176058+0000", 2024-04-26T01:08:39.100 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016219499999999999 2024-04-26T01:08:39.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.100 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.100 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.176085+0000", 2024-04-26T01:08:39.100 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7566e-05 2024-04-26T01:08:39.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.100 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.100 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.182118+0000", 2024-04-26T01:08:39.100 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0060333349999999999 2024-04-26T01:08:39.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.100 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.101 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.182211+0000", 2024-04-26T01:08:39.101 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2232999999999996e-05 2024-04-26T01:08:39.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.101 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.101 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/44 ECSubWrite(tid=353, reqid=osd.4.0:353, at_version=60'83, trim_to=0'0, roll_forward_to=59'79))", 2024-04-26T01:08:39.101 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.186337+0000", 2024-04-26T01:08:39.101 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 12.848659786000001, 2024-04-26T01:08:39.101 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0065831279999999997, 2024-04-26T01:08:39.101 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.101 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.101 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.101 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.102 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.102 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186337+0000", 2024-04-26T01:08:39.102 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.102 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.102 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186337+0000", 2024-04-26T01:08:39.102 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.102 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.102 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186339+0000", 2024-04-26T01:08:39.102 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.08e-06 2024-04-26T01:08:39.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.103 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.103 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186388+0000", 2024-04-26T01:08:39.103 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9116000000000002e-05 2024-04-26T01:08:39.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.103 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.103 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186438+0000", 2024-04-26T01:08:39.103 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9586000000000003e-05 2024-04-26T01:08:39.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.103 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.103 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186463+0000", 2024-04-26T01:08:39.103 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4859999999999999e-05 2024-04-26T01:08:39.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.104 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.104 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186608+0000", 2024-04-26T01:08:39.104 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000145162 2024-04-26T01:08:39.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.104 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.104 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186632+0000", 2024-04-26T01:08:39.104 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4179999999999999e-05 2024-04-26T01:08:39.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.104 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.104 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.192817+0000", 2024-04-26T01:08:39.104 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061849080000000002 2024-04-26T01:08:39.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.105 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.105 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.105 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.192920+0000", 2024-04-26T01:08:39.105 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000103236 2024-04-26T01:08:39.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.105 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.105 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.105 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 60/44 ECSubWrite(tid=49, reqid=osd.9.0:49, at_version=60'58, trim_to=0'0, roll_forward_to=59'56))", 2024-04-26T01:08:39.105 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.187846+0000", 2024-04-26T01:08:39.105 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 12.847151412000001, 2024-04-26T01:08:39.105 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0058844539999999999, 2024-04-26T01:08:39.106 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.106 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.106 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.106 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.106 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.106 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187846+0000", 2024-04-26T01:08:39.106 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.106 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.106 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187846+0000", 2024-04-26T01:08:39.106 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.107 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.107 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187846+0000", 2024-04-26T01:08:39.107 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7000000000000004e-07 2024-04-26T01:08:39.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.107 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.107 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.107 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187906+0000", 2024-04-26T01:08:39.107 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9246999999999997e-05 2024-04-26T01:08:39.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.107 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.107 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.107 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187998+0000", 2024-04-26T01:08:39.107 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2491999999999995e-05 2024-04-26T01:08:39.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.108 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.108 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.108 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.188021+0000", 2024-04-26T01:08:39.108 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2484e-05 2024-04-26T01:08:39.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.108 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.108 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.108 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.188184+0000", 2024-04-26T01:08:39.108 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016372500000000001 2024-04-26T01:08:39.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.108 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.108 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.108 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.188212+0000", 2024-04-26T01:08:39.109 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7963e-05 2024-04-26T01:08:39.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.109 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.109 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.193623+0000", 2024-04-26T01:08:39.109 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0054108280000000003 2024-04-26T01:08:39.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.109 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.109 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.193730+0000", 2024-04-26T01:08:39.109 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107045 2024-04-26T01:08:39.109 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.109 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.109 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.110 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 60/44 ECSubWrite(tid=52, reqid=osd.9.0:52, at_version=60'62, trim_to=0'0, roll_forward_to=60'60))", 2024-04-26T01:08:39.110 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.205252+0000", 2024-04-26T01:08:39.110 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 12.829745183, 2024-04-26T01:08:39.110 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.006970324, 2024-04-26T01:08:39.110 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.110 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.110 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.110 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.110 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.110 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205252+0000", 2024-04-26T01:08:39.110 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.111 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.111 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205252+0000", 2024-04-26T01:08:39.111 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.111 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.111 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205254+0000", 2024-04-26T01:08:39.111 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8169999999999999e-06 2024-04-26T01:08:39.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.111 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.111 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205305+0000", 2024-04-26T01:08:39.111 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.1619000000000002e-05 2024-04-26T01:08:39.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.112 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.112 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.112 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205912+0000", 2024-04-26T01:08:39.112 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00060660599999999999 2024-04-26T01:08:39.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.112 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.112 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.112 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205940+0000", 2024-04-26T01:08:39.112 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7596000000000001e-05 2024-04-26T01:08:39.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.112 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.112 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.112 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.206168+0000", 2024-04-26T01:08:39.113 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00022865200000000001 2024-04-26T01:08:39.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.113 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.113 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.113 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.206205+0000", 2024-04-26T01:08:39.113 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6439000000000001e-05 2024-04-26T01:08:39.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.113 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.113 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.114 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.212015+0000", 2024-04-26T01:08:39.114 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0058099400000000004 2024-04-26T01:08:39.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.114 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.114 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.114 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.212222+0000", 2024-04-26T01:08:39.114 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000207655 2024-04-26T01:08:39.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.114 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.114 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.114 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 71/44 ECSubWrite(tid=72, reqid=client.4521.0:729, at_version=71'79, trim_to=0'0, roll_forward_to=70'78))", 2024-04-26T01:08:39.115 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.108579+0000", 2024-04-26T01:08:39.115 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 1.9264179769999998, 2024-04-26T01:08:39.115 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0058525290000000004, 2024-04-26T01:08:39.115 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.115 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.115 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.115 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.115 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.116 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.116 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.108579+0000", 2024-04-26T01:08:39.116 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.116 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.116 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.116 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.108579+0000", 2024-04-26T01:08:39.116 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.116 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.116 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.116 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.108580+0000", 2024-04-26T01:08:39.117 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.2300000000000002e-07 2024-04-26T01:08:39.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.117 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.117 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.109172+0000", 2024-04-26T01:08:39.117 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00059268199999999997 2024-04-26T01:08:39.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.117 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.117 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.109338+0000", 2024-04-26T01:08:39.117 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000165802 2024-04-26T01:08:39.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.117 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.118 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.109363+0000", 2024-04-26T01:08:39.118 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4732999999999999e-05 2024-04-26T01:08:39.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.118 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.118 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.118 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.109489+0000", 2024-04-26T01:08:39.118 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000126289 2024-04-26T01:08:39.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.118 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.118 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.119 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.109504+0000", 2024-04-26T01:08:39.119 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5217e-05 2024-04-26T01:08:39.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.119 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.119 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.114323+0000", 2024-04-26T01:08:39.119 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0048184179999999997 2024-04-26T01:08:39.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.119 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.119 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.114432+0000", 2024-04-26T01:08:39.119 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000108965 2024-04-26T01:08:39.119 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.119 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.119 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.120 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.120 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 71/44 ECSubWrite(tid=73, reqid=osd.9.0:73, at_version=71'81, trim_to=0'0, roll_forward_to=70'78))", 2024-04-26T01:08:39.120 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.110054+0000", 2024-04-26T01:08:39.120 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 1.924943356, 2024-04-26T01:08:39.120 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0070291959999999997, 2024-04-26T01:08:39.120 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.120 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.120 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.120 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.120 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.120 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.120 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110054+0000", 2024-04-26T01:08:39.120 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.121 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.121 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.121 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110054+0000", 2024-04-26T01:08:39.121 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.121 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.121 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.121 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110055+0000", 2024-04-26T01:08:39.121 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.48e-06 2024-04-26T01:08:39.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.121 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.121 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.121 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110105+0000", 2024-04-26T01:08:39.121 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9332999999999997e-05 2024-04-26T01:08:39.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.122 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.122 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110193+0000", 2024-04-26T01:08:39.122 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8938999999999997e-05 2024-04-26T01:08:39.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.122 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.122 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110214+0000", 2024-04-26T01:08:39.122 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0939999999999999e-05 2024-04-26T01:08:39.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.123 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.123 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110505+0000", 2024-04-26T01:08:39.123 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00029090499999999999 2024-04-26T01:08:39.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.123 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.123 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110521+0000", 2024-04-26T01:08:39.123 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5855999999999999e-05 2024-04-26T01:08:39.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.123 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.124 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.117043+0000", 2024-04-26T01:08:39.124 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0065219570000000001 2024-04-26T01:08:39.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.124 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.124 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.124 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.117083+0000", 2024-04-26T01:08:39.124 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.9786000000000002e-05 2024-04-26T01:08:39.124 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.124 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.124 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.124 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.125 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 71/44 ECSubWrite(tid=75, reqid=client.4521.0:731, at_version=71'83, trim_to=0'0, roll_forward_to=71'79))", 2024-04-26T01:08:39.125 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.124820+0000", 2024-04-26T01:08:39.125 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 1.9101769310000001, 2024-04-26T01:08:39.125 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0069566530000000001, 2024-04-26T01:08:39.125 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.125 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.125 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.125 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.125 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.125 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.125 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.124820+0000", 2024-04-26T01:08:39.126 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.126 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.126 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.126 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.124820+0000", 2024-04-26T01:08:39.126 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.126 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.126 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.126 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.124822+0000", 2024-04-26T01:08:39.126 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8649999999999999e-06 2024-04-26T01:08:39.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.127 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.127 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.127 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.125176+0000", 2024-04-26T01:08:39.127 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00035433799999999999 2024-04-26T01:08:39.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.127 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.127 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.127 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.125244+0000", 2024-04-26T01:08:39.127 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7162999999999994e-05 2024-04-26T01:08:39.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.128 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.128 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.128 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.125258+0000", 2024-04-26T01:08:39.128 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4418e-05 2024-04-26T01:08:39.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.128 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.128 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.128 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.125406+0000", 2024-04-26T01:08:39.128 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014758499999999999 2024-04-26T01:08:39.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.129 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.129 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.129 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.125435+0000", 2024-04-26T01:08:39.129 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.932e-05 2024-04-26T01:08:39.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.129 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.129 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.129 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.131716+0000", 2024-04-26T01:08:39.129 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062809900000000002 2024-04-26T01:08:39.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.129 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.130 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.130 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.131777+0000", 2024-04-26T01:08:39.130 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0974000000000003e-05 2024-04-26T01:08:39.130 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.130 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.130 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.130 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.130 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.130 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 71/44 ECSubWrite(tid=76, reqid=client.4521.0:732, at_version=71'84, trim_to=0'0, roll_forward_to=71'81))", 2024-04-26T01:08:39.130 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.127563+0000", 2024-04-26T01:08:39.130 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 1.90743365, 2024-04-26T01:08:39.130 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066958829999999997, 2024-04-26T01:08:39.130 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:39.130 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:39.130 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:39.131 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:39.131 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.131 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:39.131 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127563+0000", 2024-04-26T01:08:39.131 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.131 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.131 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:39.131 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127563+0000", 2024-04-26T01:08:39.131 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:39.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.131 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.131 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:39.131 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127565+0000", 2024-04-26T01:08:39.131 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.17e-06 2024-04-26T01:08:39.132 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.132 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.132 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:39.132 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127666+0000", 2024-04-26T01:08:39.132 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000100997 2024-04-26T01:08:39.132 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.132 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.132 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:39.132 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127710+0000", 2024-04-26T01:08:39.132 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.4496000000000002e-05 2024-04-26T01:08:39.132 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.132 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.132 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:39.132 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127722+0000", 2024-04-26T01:08:39.132 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2252000000000001e-05 2024-04-26T01:08:39.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.133 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.133 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:39.133 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127954+0000", 2024-04-26T01:08:39.133 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00023175299999999999 2024-04-26T01:08:39.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.133 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.133 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:39.133 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127977+0000", 2024-04-26T01:08:39.133 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3093000000000001e-05 2024-04-26T01:08:39.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.133 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.133 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:39.134 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.134210+0000", 2024-04-26T01:08:39.134 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062328790000000002 2024-04-26T01:08:39.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:39.134 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:39.134 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:39.134 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.134259+0000", 2024-04-26T01:08:39.134 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9243000000000003e-05 2024-04-26T01:08:39.134 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.134 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.134 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.134 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:39.135 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:39.135 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:39.136 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:39.136 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:39.137 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:39.137 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:39.137 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:08:39.138 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:39.138 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:39.138 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:39.139 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:39.139 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:39.132+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:39.161 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:39.161 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:39.194 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:39.195 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:39.228 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:39.229 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:39.237 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:39.235+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:39.246 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-04-26T01:08:39.247 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.8 to 1 (10000) 2024-04-26T01:08:39.251 INFO:tasks.rados.rados.0.smithi164.stdout:304: done (2 left) 2024-04-26T01:08:39.251 INFO:tasks.rados.rados.0.smithi164.stdout:305: done (1 left) 2024-04-26T01:08:39.251 INFO:tasks.rados.rados.0.smithi164.stdout:306: done (0 left) 2024-04-26T01:08:39.251 INFO:tasks.rados.rados.0.smithi164.stdout:307: setattr oid 22 current snap is 21 2024-04-26T01:08:39.253 INFO:tasks.rados.rados.0.smithi164.stdout:308: read oid 21 snap 20 2024-04-26T01:08:39.253 INFO:tasks.rados.rados.0.smithi164.stdout:308: expect (ObjNum 88 snap 6 seq_num 88) 2024-04-26T01:08:39.253 INFO:tasks.rados.rados.0.smithi164.stdout:309: read oid 44 snap 19 2024-04-26T01:08:39.253 INFO:tasks.rados.rados.0.smithi164.stdout:309: expect (ObjNum 113 snap 14 seq_num 113) 2024-04-26T01:08:39.253 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 41 current snap is 21 2024-04-26T01:08:39.253 INFO:tasks.rados.rados.0.smithi164.stdout:310: seq_num 135 ranges {688128=720896,1409024=622592,2031616=688128,2719744=245760} 2024-04-26T01:08:39.265 INFO:tasks.rados.rados.0.smithi164.stdout:310: writing smithi16428812-41 from 688128 to 1409024 tid 1 2024-04-26T01:08:39.270 INFO:tasks.rados.rados.0.smithi164.stdout:310: writing smithi16428812-41 from 1409024 to 2031616 tid 2 2024-04-26T01:08:39.271 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:39.271 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:39.277 INFO:tasks.rados.rados.0.smithi164.stdout:310: writing smithi16428812-41 from 2031616 to 2719744 tid 3 2024-04-26T01:08:39.280 INFO:tasks.rados.rados.0.smithi164.stdout:310: writing smithi16428812-41 from 2719744 to 2965504 tid 4 2024-04-26T01:08:39.280 INFO:tasks.rados.rados.0.smithi164.stdout:311: read oid 14 snap 20 2024-04-26T01:08:39.280 INFO:tasks.rados.rados.0.smithi164.stdout:311: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:08:39.280 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 17 current snap is 21 2024-04-26T01:08:39.280 INFO:tasks.rados.rados.0.smithi164.stdout:312: seq_num 136 ranges {81920=737280,819200=770048,1589248=540672,2129920=245760} 2024-04-26T01:08:39.288 INFO:tasks.rados.rados.0.smithi164.stdout:312: writing smithi16428812-17 from 81920 to 819200 tid 1 2024-04-26T01:08:39.290 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-04-26T01:08:39.296 INFO:tasks.rados.rados.0.smithi164.stdout:312: writing smithi16428812-17 from 819200 to 1589248 tid 2 2024-04-26T01:08:39.302 INFO:tasks.rados.rados.0.smithi164.stdout:312: writing smithi16428812-17 from 1589248 to 2129920 tid 3 2024-04-26T01:08:39.304 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:39.304 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:39.305 INFO:tasks.rados.rados.0.smithi164.stdout:312: writing smithi16428812-17 from 2129920 to 2375680 tid 4 2024-04-26T01:08:39.306 INFO:tasks.rados.rados.0.smithi164.stdout:313: read oid 39 snap 20 2024-04-26T01:08:39.306 INFO:tasks.rados.rados.0.smithi164.stdout:313: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-26T01:08:39.306 INFO:tasks.rados.rados.0.smithi164.stdout:314: copy_from oid 42 from oid 18 current snap is 21 2024-04-26T01:08:39.306 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 22 v 76 (ObjNum 95 snap 8 seq_num 95) dirty exists 2024-04-26T01:08:39.306 INFO:tasks.rados.rados.0.smithi164.stdout:308: expect (ObjNum 88 snap 6 seq_num 88) 2024-04-26T01:08:39.328 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:39.328 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:39.328 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:08:39.328 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:39.334 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:08:39.335 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:08:39.338 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:39.336+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:39.343 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:08:39.356 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:39.357 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:39.383 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:39.383 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:39.406 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:39.404+0000 7f3fb3fff640 0 -- 172.21.15.8:0/38849089 >> [v2:172.21.15.116:6800/2158650188,v1:172.21.15.116:6801/2158650188] conn(0x7f3fbc227450 msgr2=0x7f3fbc229840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:08:39.412 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:39.438 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:39.438 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:39.439 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:39.439+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:39.452 INFO:tasks.rados.rados.0.smithi164.stdout:310: finishing write tid 1 to smithi16428812-41 2024-04-26T01:08:39.453 INFO:tasks.rados.rados.0.smithi164.stdout:310: finishing write tid 2 to smithi16428812-41 2024-04-26T01:08:39.453 INFO:tasks.rados.rados.0.smithi164.stdout:310: finishing write tid 3 to smithi16428812-41 2024-04-26T01:08:39.453 INFO:tasks.rados.rados.0.smithi164.stdout:310: finishing write tid 4 to smithi16428812-41 2024-04-26T01:08:39.453 INFO:tasks.rados.rados.0.smithi164.stdout:310: finishing write tid 5 to smithi16428812-41 2024-04-26T01:08:39.453 INFO:tasks.rados.rados.0.smithi164.stdout:311: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:08:39.460 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:39.461 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:39.475 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:08:39.483 INFO:tasks.rados.rados.0.smithi164.stdout:310: finishing write tid 6 to smithi16428812-41 2024-04-26T01:08:39.483 INFO:tasks.rados.rados.0.smithi164.stdout:310: oid 41 updating version 0 to 68 2024-04-26T01:08:39.484 INFO:tasks.rados.rados.0.smithi164.stdout:310: oid 41 version 68 is already newer than 66 2024-04-26T01:08:39.484 INFO:tasks.rados.rados.0.smithi164.stdout:310: oid 41 updating version 68 to 69 2024-04-26T01:08:39.484 INFO:tasks.rados.rados.0.smithi164.stdout:310: oid 41 version 69 is already newer than 67 2024-04-26T01:08:39.484 INFO:tasks.rados.rados.0.smithi164.stdout:310: oid 41 updating version 69 to 70 2024-04-26T01:08:39.484 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 41 v 70 (ObjNum 135 snap 21 seq_num 135) dirty exists 2024-04-26T01:08:39.484 INFO:tasks.rados.rados.0.smithi164.stdout:310: left oid 41 (ObjNum 135 snap 21 seq_num 135) 2024-04-26T01:08:39.484 INFO:tasks.rados.rados.0.smithi164.stdout:313: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-26T01:08:39.497 INFO:tasks.rados.rados.0.smithi164.stdout:312: finishing write tid 1 to smithi16428812-17 2024-04-26T01:08:39.497 INFO:tasks.rados.rados.0.smithi164.stdout:312: finishing write tid 2 to smithi16428812-17 2024-04-26T01:08:39.497 INFO:tasks.rados.rados.0.smithi164.stdout:312: finishing write tid 3 to smithi16428812-17 2024-04-26T01:08:39.497 INFO:tasks.rados.rados.0.smithi164.stdout:312: finishing write tid 4 to smithi16428812-17 2024-04-26T01:08:39.497 INFO:tasks.rados.rados.0.smithi164.stdout:312: finishing write tid 5 to smithi16428812-17 2024-04-26T01:08:39.497 INFO:tasks.rados.rados.0.smithi164.stdout:314: finishing copy_from racing read to smithi16428812-42 2024-04-26T01:08:39.497 INFO:tasks.rados.rados.0.smithi164.stdout:312: finishing write tid 6 to smithi16428812-17 2024-04-26T01:08:39.497 INFO:tasks.rados.rados.0.smithi164.stdout:312: oid 17 updating version 0 to 38 2024-04-26T01:08:39.497 INFO:tasks.rados.rados.0.smithi164.stdout:312: oid 17 updating version 38 to 40 2024-04-26T01:08:39.497 INFO:tasks.rados.rados.0.smithi164.stdout:312: oid 17 updating version 40 to 41 2024-04-26T01:08:39.497 INFO:tasks.rados.rados.0.smithi164.stdout:312: oid 17 version 41 is already newer than 39 2024-04-26T01:08:39.498 INFO:tasks.rados.rados.0.smithi164.stdout:312: oid 17 version 41 is already newer than 37 2024-04-26T01:08:39.498 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 17 v 41 (ObjNum 136 snap 21 seq_num 136) dirty exists 2024-04-26T01:08:39.498 INFO:tasks.rados.rados.0.smithi164.stdout:312: left oid 17 (ObjNum 136 snap 21 seq_num 136) 2024-04-26T01:08:39.498 INFO:tasks.rados.rados.0.smithi164.stdout:314: finishing copy_from to smithi16428812-42 2024-04-26T01:08:39.498 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 42 v 71 (ObjNum 111 snap 14 seq_num 111) dirty exists 2024-04-26T01:08:39.498 INFO:tasks.rados.rados.0.smithi164.stdout:307: done (7 left) 2024-04-26T01:08:39.498 INFO:tasks.rados.rados.0.smithi164.stdout:308: done (6 left) 2024-04-26T01:08:39.499 INFO:tasks.rados.rados.0.smithi164.stdout:310: done (5 left) 2024-04-26T01:08:39.499 INFO:tasks.rados.rados.0.smithi164.stdout:311: done (4 left) 2024-04-26T01:08:39.499 INFO:tasks.rados.rados.0.smithi164.stdout:312: done (3 left) 2024-04-26T01:08:39.499 INFO:tasks.rados.rados.0.smithi164.stdout:313: done (2 left) 2024-04-26T01:08:39.499 INFO:tasks.rados.rados.0.smithi164.stdout:314: done (1 left) 2024-04-26T01:08:39.499 INFO:tasks.rados.rados.0.smithi164.stdout:315: rmattr oid 6 current snap is 21 2024-04-26T01:08:39.499 INFO:tasks.rados.rados.0.smithi164.stdout:315: done (1 left) 2024-04-26T01:08:39.499 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 41 current snap is 21 2024-04-26T01:08:39.499 INFO:tasks.rados.rados.0.smithi164.stdout:316: seq_num 137 ranges {2965504=753664,3719168=786432,4505600=655360,5160960=114688} 2024-04-26T01:08:39.541 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:39.539+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:39.553 INFO:tasks.rados.rados.0.smithi164.stdout:316: writing smithi16428812-41 from 2965504 to 3719168 tid 1 2024-04-26T01:08:39.562 INFO:tasks.rados.rados.0.smithi164.stdout:316: writing smithi16428812-41 from 3719168 to 4505600 tid 2 2024-04-26T01:08:39.569 INFO:tasks.rados.rados.0.smithi164.stdout:316: writing smithi16428812-41 from 4505600 to 5160960 tid 3 2024-04-26T01:08:39.570 INFO:tasks.rados.rados.0.smithi164.stdout:316: writing smithi16428812-41 from 5160960 to 5275648 tid 4 2024-04-26T01:08:39.570 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 11 current snap is 21 2024-04-26T01:08:39.570 INFO:tasks.rados.rados.0.smithi164.stdout:317: seq_num 138 ranges {0=770048,770048=540672,1310720=753664,2064384=262144} 2024-04-26T01:08:39.577 INFO:tasks.rados.rados.0.smithi164.stdout:317: writing smithi16428812-11 from 0 to 770048 tid 1 2024-04-26T01:08:39.581 INFO:tasks.rados.rados.0.smithi164.stdout:317: writing smithi16428812-11 from 770048 to 1310720 tid 2 2024-04-26T01:08:39.587 INFO:tasks.rados.rados.0.smithi164.stdout:317: writing smithi16428812-11 from 1310720 to 2064384 tid 3 2024-04-26T01:08:39.590 INFO:tasks.rados.rados.0.smithi164.stdout:317: writing smithi16428812-11 from 2064384 to 2326528 tid 4 2024-04-26T01:08:39.590 INFO:tasks.rados.rados.0.smithi164.stdout:318: delete oid 1 current snap is 21 2024-04-26T01:08:39.590 INFO:tasks.rados.rados.0.smithi164.stdout:316: finishing write tid 1 to smithi16428812-41 2024-04-26T01:08:39.590 INFO:tasks.rados.rados.0.smithi164.stdout:316: finishing write tid 2 to smithi16428812-41 2024-04-26T01:08:39.590 INFO:tasks.rados.rados.0.smithi164.stdout:316: finishing write tid 3 to smithi16428812-41 2024-04-26T01:08:39.590 INFO:tasks.rados.rados.0.smithi164.stdout:316: finishing write tid 4 to smithi16428812-41 2024-04-26T01:08:39.590 INFO:tasks.rados.rados.0.smithi164.stdout:316: finishing write tid 5 to smithi16428812-41 2024-04-26T01:08:39.591 INFO:tasks.rados.rados.0.smithi164.stdout:316: finishing write tid 6 to smithi16428812-41 2024-04-26T01:08:39.591 INFO:tasks.rados.rados.0.smithi164.stdout:316: oid 41 updating version 0 to 75 2024-04-26T01:08:39.591 INFO:tasks.rados.rados.0.smithi164.stdout:316: oid 41 version 75 is already newer than 74 2024-04-26T01:08:39.591 INFO:tasks.rados.rados.0.smithi164.stdout:316: oid 41 version 75 is already newer than 72 2024-04-26T01:08:39.591 INFO:tasks.rados.rados.0.smithi164.stdout:316: oid 41 version 75 is already newer than 71 2024-04-26T01:08:39.591 INFO:tasks.rados.rados.0.smithi164.stdout:316: oid 41 version 75 is already newer than 73 2024-04-26T01:08:39.591 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 41 v 75 (ObjNum 137 snap 21 seq_num 137) dirty exists 2024-04-26T01:08:39.591 INFO:tasks.rados.rados.0.smithi164.stdout:316: left oid 41 (ObjNum 137 snap 21 seq_num 137) 2024-04-26T01:08:39.596 INFO:tasks.rados.rados.0.smithi164.stdout:316: done (3 left) 2024-04-26T01:08:39.596 INFO:tasks.rados.rados.0.smithi164.stdout:318: done (2 left) 2024-04-26T01:08:39.596 INFO:tasks.rados.rados.0.smithi164.stdout:319: snap_remove snap 20 2024-04-26T01:08:39.642 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:39.637+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:39.651 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:39.651 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:39.665 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:08:39.743 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:39.741+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:39.797 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:39.797 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:39.811 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:08:39.844 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:39.840+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:39.931 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:39.931 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:39.945 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:08:39.946 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:39.945+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:40.045 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:40.043+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:40.068 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:40.068 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:40.083 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:08:40.147 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:40.143+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:40.209 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:40.210 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:40.225 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:08:40.248 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:40.245+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:40.254 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.9 to 1 (10000) 2024-04-26T01:08:40.259 INFO:tasks.rados.rados.0.smithi164.stdout:319: done (2 left) 2024-04-26T01:08:40.259 INFO:tasks.rados.rados.0.smithi164.stdout:320: rollback oid 20 current snap is 21 2024-04-26T01:08:40.259 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 20 to 19 2024-04-26T01:08:40.259 INFO:tasks.rados.rados.0.smithi164.stdout:317: finishing write tid 1 to smithi16428812-11 2024-04-26T01:08:40.259 INFO:tasks.rados.rados.0.smithi164.stdout:317: finishing write tid 2 to smithi16428812-11 2024-04-26T01:08:40.259 INFO:tasks.rados.rados.0.smithi164.stdout:317: finishing write tid 3 to smithi16428812-11 2024-04-26T01:08:40.259 INFO:tasks.rados.rados.0.smithi164.stdout:317: finishing write tid 4 to smithi16428812-11 2024-04-26T01:08:40.259 INFO:tasks.rados.rados.0.smithi164.stdout:309: expect (ObjNum 113 snap 14 seq_num 113) 2024-04-26T01:08:40.285 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-04-26T01:08:40.348 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:40.347+0000 7ff2871de640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 32810) UID: 0 2024-04-26T01:08:40.354 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:40.355 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:40.369 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:08:40.450 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:40.449+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:40.521 INFO:tasks.rados.rados.0.smithi164.stdout:317: finishing write tid 5 to smithi16428812-11 2024-04-26T01:08:40.521 INFO:tasks.rados.rados.0.smithi164.stdout:317: finishing write tid 6 to smithi16428812-11 2024-04-26T01:08:40.522 INFO:tasks.rados.rados.0.smithi164.stdout:317: oid 11 updating version 0 to 33 2024-04-26T01:08:40.522 INFO:tasks.rados.rados.0.smithi164.stdout:317: oid 11 updating version 33 to 34 2024-04-26T01:08:40.522 INFO:tasks.rados.rados.0.smithi164.stdout:317: oid 11 updating version 34 to 36 2024-04-26T01:08:40.522 INFO:tasks.rados.rados.0.smithi164.stdout:317: oid 11 updating version 36 to 37 2024-04-26T01:08:40.522 INFO:tasks.rados.rados.0.smithi164.stdout:317: oid 11 version 37 is already newer than 35 2024-04-26T01:08:40.522 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 11 v 37 (ObjNum 138 snap 21 seq_num 138) dirty exists 2024-04-26T01:08:40.522 INFO:tasks.rados.rados.0.smithi164.stdout:317: left oid 11 (ObjNum 138 snap 21 seq_num 138) 2024-04-26T01:08:40.522 INFO:tasks.rados.rados.0.smithi164.stdout:309: done (2 left) 2024-04-26T01:08:40.522 INFO:tasks.rados.rados.0.smithi164.stdout:317: done (1 left) 2024-04-26T01:08:40.522 INFO:tasks.rados.rados.0.smithi164.stdout:321: read oid 28 snap -1 2024-04-26T01:08:40.522 INFO:tasks.rados.rados.0.smithi164.stdout:321: expect (ObjNum 108 snap 13 seq_num 108) 2024-04-26T01:08:40.522 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 21 current snap is 21 2024-04-26T01:08:40.522 INFO:tasks.rados.rados.0.smithi164.stdout:320: finishing rollback tid 0 to smithi16428812-20 2024-04-26T01:08:40.522 INFO:tasks.rados.rados.0.smithi164.stdout:320: finishing rollback tid 1 to smithi16428812-20 2024-04-26T01:08:40.523 INFO:tasks.rados.rados.0.smithi164.stdout:322: seq_num 140 ranges {2719744=802816,3522560=425984,3948544=720896,4669440=409600} 2024-04-26T01:08:40.535 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:40.535 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:40.547 INFO:tasks.rados.rados.0.smithi164.stdout:322: writing smithi16428812-21 from 2719744 to 3522560 tid 1 2024-04-26T01:08:40.550 INFO:tasks.rados.rados.0.smithi164.stdout:322: writing smithi16428812-21 from 3522560 to 3948544 tid 2 2024-04-26T01:08:40.551 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:40.547+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:40.555 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:08:40.556 INFO:tasks.rados.rados.0.smithi164.stdout:322: writing smithi16428812-21 from 3948544 to 4669440 tid 3 2024-04-26T01:08:40.559 INFO:tasks.rados.rados.0.smithi164.stdout:322: writing smithi16428812-21 from 4669440 to 5079040 tid 4 2024-04-26T01:08:40.559 INFO:tasks.rados.rados.0.smithi164.stdout:323: copy_from oid 3 from oid 5 current snap is 21 2024-04-26T01:08:40.560 INFO:tasks.rados.rados.0.smithi164.stdout:321: expect (ObjNum 108 snap 13 seq_num 108) 2024-04-26T01:08:40.651 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:40.651+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:40.676 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:40.676 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:40.690 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:08:40.752 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:40.751+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:40.772 INFO:tasks.rados.rados.0.smithi164.stdout:322: finishing write tid 1 to smithi16428812-21 2024-04-26T01:08:40.772 INFO:tasks.rados.rados.0.smithi164.stdout:322: finishing write tid 2 to smithi16428812-21 2024-04-26T01:08:40.772 INFO:tasks.rados.rados.0.smithi164.stdout:322: finishing write tid 3 to smithi16428812-21 2024-04-26T01:08:40.772 INFO:tasks.rados.rados.0.smithi164.stdout:322: finishing write tid 4 to smithi16428812-21 2024-04-26T01:08:40.773 INFO:tasks.rados.rados.0.smithi164.stdout:322: finishing write tid 5 to smithi16428812-21 2024-04-26T01:08:40.773 INFO:tasks.rados.rados.0.smithi164.stdout:322: finishing write tid 6 to smithi16428812-21 2024-04-26T01:08:40.773 INFO:tasks.rados.rados.0.smithi164.stdout:322: oid 21 updating version 0 to 55 2024-04-26T01:08:40.773 INFO:tasks.rados.rados.0.smithi164.stdout:322: oid 21 updating version 55 to 56 2024-04-26T01:08:40.773 INFO:tasks.rados.rados.0.smithi164.stdout:322: oid 21 version 56 is already newer than 53 2024-04-26T01:08:40.773 INFO:tasks.rados.rados.0.smithi164.stdout:322: oid 21 version 56 is already newer than 54 2024-04-26T01:08:40.773 INFO:tasks.rados.rados.0.smithi164.stdout:322: oid 21 version 56 is already newer than 52 2024-04-26T01:08:40.773 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 21 v 56 (ObjNum 140 snap 21 seq_num 140) dirty exists 2024-04-26T01:08:40.773 INFO:tasks.rados.rados.0.smithi164.stdout:322: left oid 21 (ObjNum 140 snap 21 seq_num 140) 2024-04-26T01:08:40.773 INFO:tasks.rados.rados.0.smithi164.stdout:323: finishing copy_from racing read to smithi16428812-3 2024-04-26T01:08:40.773 INFO:tasks.rados.rados.0.smithi164.stdout:323: finishing copy_from to smithi16428812-3 2024-04-26T01:08:40.773 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 3 v 32 (ObjNum 127 snap 19 seq_num 127) dirty exists 2024-04-26T01:08:40.773 INFO:tasks.rados.rados.0.smithi164.stdout:321: done (3 left) 2024-04-26T01:08:40.773 INFO:tasks.rados.rados.0.smithi164.stdout:322: done (2 left) 2024-04-26T01:08:40.774 INFO:tasks.rados.rados.0.smithi164.stdout:323: done (1 left) 2024-04-26T01:08:40.774 INFO:tasks.rados.rados.0.smithi164.stdout:324: delete oid 22 current snap is 21 2024-04-26T01:08:40.777 INFO:tasks.rados.rados.0.smithi164.stdout:324: done (1 left) 2024-04-26T01:08:40.778 INFO:tasks.rados.rados.0.smithi164.stdout:325: delete oid 33 current snap is 21 2024-04-26T01:08:40.785 INFO:tasks.rados.rados.0.smithi164.stdout:325: done (1 left) 2024-04-26T01:08:40.785 INFO:tasks.rados.rados.0.smithi164.stdout:326: delete oid 28 current snap is 21 2024-04-26T01:08:40.786 INFO:tasks.rados.rados.0.smithi164.stdout:320: finishing rollback tid 2 to smithi16428812-20 2024-04-26T01:08:40.786 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 20 v 14 (ObjNum 69 snap 4 seq_num 69) dirty exists 2024-04-26T01:08:40.790 INFO:tasks.rados.rados.0.smithi164.stdout:320: done (1 left) 2024-04-26T01:08:40.790 INFO:tasks.rados.rados.0.smithi164.stdout:326: done (0 left) 2024-04-26T01:08:40.790 INFO:tasks.rados.rados.0.smithi164.stdout:327: rollback oid 33 current snap is 21 2024-04-26T01:08:40.790 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 33 to 19 2024-04-26T01:08:40.790 INFO:tasks.rados.rados.0.smithi164.stdout:328: snap_create 2024-04-26T01:08:40.796 INFO:tasks.rados.rados.0.smithi164.stdout:327: finishing rollback tid 1 to smithi16428812-33 2024-04-26T01:08:40.797 INFO:tasks.rados.rados.0.smithi164.stdout:327: finishing rollback tid 2 to smithi16428812-33 2024-04-26T01:08:40.797 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 33 v 89 (ObjNum 126 snap 19 seq_num 126) dirty exists 2024-04-26T01:08:40.853 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:40.851+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:40.859 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:40.859 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:40.875 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:08:40.954 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:40.953+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:41.040 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:41.040 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:41.041 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.10 to 1 (10000) 2024-04-26T01:08:41.046 INFO:tasks.rados.rados.0.smithi164.stdout:327: done (1 left) 2024-04-26T01:08:41.046 INFO:tasks.rados.rados.0.smithi164.stdout:328: done (0 left) 2024-04-26T01:08:41.046 INFO:tasks.rados.rados.0.smithi164.stdout:329: snap_remove snap 21 2024-04-26T01:08:41.054 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:41.049+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:41.056 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:08:41.065 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-04-26T01:08:41.155 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:41.152+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:41.179 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:41.179 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:41.193 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:08:41.256 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:41.252+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:41.285 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-26T01:08:41.319 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:41.319 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:41.333 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:08:41.357 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:41.352+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:41.458 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:41.458 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:41.458 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:41.453+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:41.460 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:08:41.473 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:08:41.559 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:41.555+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:41.624 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:41.624 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:41.628 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:41.628 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:41.629 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-04-26T01:08:41.629 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:41.643 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:08:41.649 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:41.649 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:41.660 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:41.659+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:41.674 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:41.675 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:41.700 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:41.700 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:41.723 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:41.724 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:41.750 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:41.750 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:41.755 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:41.756 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:41.756 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-04-26T01:08:41.756 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-04-26T01:08:41.756 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:41.760 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:41.757+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:41.769 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:08:41.780 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:41.780 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:41.810 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:41.810 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:41.838 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:41.838 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:41.861 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:41.859+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:41.869 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:41.870 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:41.908 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:41.909 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:41.910 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:41.910 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:41.910 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:41.910 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:41.910 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.910 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.7s3 33/30 ECSubWrite(tid=9, reqid=client.4521.0:38, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:41.910 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.325556+0000", 2024-04-26T01:08:41.910 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 42.580738922000002, 2024-04-26T01:08:41.910 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.028941706000000001, 2024-04-26T01:08:41.911 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.911 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.911 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:41.911 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.911 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.911 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.911 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325556+0000", 2024-04-26T01:08:41.911 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.911 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.912 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.912 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.912 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325556+0000", 2024-04-26T01:08:41.912 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.912 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.912 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.912 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.912 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325557+0000", 2024-04-26T01:08:41.912 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.221e-06 2024-04-26T01:08:41.913 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.913 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.913 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.913 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325581+0000", 2024-04-26T01:08:41.913 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3502e-05 2024-04-26T01:08:41.913 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.913 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.914 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.914 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353123+0000", 2024-04-26T01:08:41.914 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.027542540000000001 2024-04-26T01:08:41.914 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.914 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.914 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.914 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353131+0000", 2024-04-26T01:08:41.915 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6909999999999997e-06 2024-04-26T01:08:41.915 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.915 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.915 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.915 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353823+0000", 2024-04-26T01:08:41.915 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00069229100000000004 2024-04-26T01:08:41.915 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.915 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.915 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:41.915 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353835+0000", 2024-04-26T01:08:41.915 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1507e-05 2024-04-26T01:08:41.915 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.915 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.915 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:41.916 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.354460+0000", 2024-04-26T01:08:41.916 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00062488599999999995 2024-04-26T01:08:41.916 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.916 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.916 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.916 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.354498+0000", 2024-04-26T01:08:41.916 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8068000000000002e-05 2024-04-26T01:08:41.916 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.916 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.916 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.916 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.916 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.916 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:45 3.6s0 3:61b5d9f4:::smithi16428812-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:41.916 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.345679+0000", 2024-04-26T01:08:41.917 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 42.560616072000002, 2024-04-26T01:08:41.917 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014982359000000001, 2024-04-26T01:08:41.917 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.917 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.917 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:41.917 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:41.917 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:41.917 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:41.917 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 45 2024-04-26T01:08:41.917 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.917 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.918 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.918 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.918 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345679+0000", 2024-04-26T01:08:41.918 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.919 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.919 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.919 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.919 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345679+0000", 2024-04-26T01:08:41.919 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.919 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.919 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.919 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.919 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345679+0000", 2024-04-26T01:08:41.919 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9599999999999999e-07 2024-04-26T01:08:41.919 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.920 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.920 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.920 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345683+0000", 2024-04-26T01:08:41.920 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0110000000000001e-06 2024-04-26T01:08:41.920 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.920 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.920 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.920 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345696+0000", 2024-04-26T01:08:41.920 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3557000000000001e-05 2024-04-26T01:08:41.920 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.920 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.920 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.920 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.345701+0000", 2024-04-26T01:08:41.921 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6190000000000002e-06 2024-04-26T01:08:41.921 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.921 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.921 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.921 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349537+0000", 2024-04-26T01:08:41.921 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0038363500000000001 2024-04-26T01:08:41.921 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.921 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.921 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:41.921 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349626+0000", 2024-04-26T01:08:41.922 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8757000000000004e-05 2024-04-26T01:08:41.922 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.922 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.922 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.922 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.358038+0000", 2024-04-26T01:08:41.922 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0084123970000000003 2024-04-26T01:08:41.922 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.922 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.922 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:41.922 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.358133+0000", 2024-04-26T01:08:41.922 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4949000000000002e-05 2024-04-26T01:08:41.922 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.922 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.923 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.923 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.360661+0000", 2024-04-26T01:08:41.923 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002528223 2024-04-26T01:08:41.923 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.923 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.923 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.923 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.923 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.923 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.fs6 33/30 ECSubWrite(tid=13, reqid=client.4521.0:53, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:41.923 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.352280+0000", 2024-04-26T01:08:41.923 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 42.554015483000001, 2024-04-26T01:08:41.923 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.017609129000000001, 2024-04-26T01:08:41.923 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.923 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.924 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:41.924 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.924 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.924 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.924 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.352280+0000", 2024-04-26T01:08:41.924 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.924 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.924 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.924 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.924 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.352280+0000", 2024-04-26T01:08:41.924 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.924 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.925 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.925 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.925 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.352280+0000", 2024-04-26T01:08:41.925 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6400000000000001e-07 2024-04-26T01:08:41.925 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.925 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.925 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.925 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.352302+0000", 2024-04-26T01:08:41.925 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1577e-05 2024-04-26T01:08:41.925 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.925 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.925 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.926 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369217+0000", 2024-04-26T01:08:41.926 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.016914825000000001 2024-04-26T01:08:41.926 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.926 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.926 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.926 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369224+0000", 2024-04-26T01:08:41.926 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9720000000000003e-06 2024-04-26T01:08:41.926 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.926 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.926 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.926 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369277+0000", 2024-04-26T01:08:41.926 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2924999999999999e-05 2024-04-26T01:08:41.927 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.927 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.927 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:41.927 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369284+0000", 2024-04-26T01:08:41.927 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0840000000000003e-06 2024-04-26T01:08:41.927 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.927 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.927 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:41.927 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369847+0000", 2024-04-26T01:08:41.927 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00056294600000000004 2024-04-26T01:08:41.927 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.927 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.927 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.927 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.369889+0000", 2024-04-26T01:08:41.928 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1936e-05 2024-04-26T01:08:41.928 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.928 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.928 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.928 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.928 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.928 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.7s3 33/30 ECSubWrite(tid=16, reqid=client.4521.0:65, at_version=33'8, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:08:41.928 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.366487+0000", 2024-04-26T01:08:41.928 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 42.539807596000003, 2024-04-26T01:08:41.928 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.079262088999999994, 2024-04-26T01:08:41.928 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.928 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.928 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:41.929 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.929 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.929 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.929 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366487+0000", 2024-04-26T01:08:41.929 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.929 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.929 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.929 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.929 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366487+0000", 2024-04-26T01:08:41.929 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.929 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.929 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.929 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.929 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366488+0000", 2024-04-26T01:08:41.929 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6700000000000002e-07 2024-04-26T01:08:41.930 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.930 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.930 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.930 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366510+0000", 2024-04-26T01:08:41.930 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1591999999999999e-05 2024-04-26T01:08:41.930 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.930 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.930 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.930 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444763+0000", 2024-04-26T01:08:41.930 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.078252639999999998 2024-04-26T01:08:41.930 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.931 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.931 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.931 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444773+0000", 2024-04-26T01:08:41.931 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0134e-05 2024-04-26T01:08:41.931 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.931 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.931 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.931 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444870+0000", 2024-04-26T01:08:41.931 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6847000000000003e-05 2024-04-26T01:08:41.931 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.931 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.932 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:41.932 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444880+0000", 2024-04-26T01:08:41.932 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0312999999999999e-05 2024-04-26T01:08:41.932 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.932 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.932 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:41.932 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.445715+0000", 2024-04-26T01:08:41.932 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00083498700000000003 2024-04-26T01:08:41.932 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.932 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.932 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.932 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.445750+0000", 2024-04-26T01:08:41.932 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4609e-05 2024-04-26T01:08:41.932 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.933 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.933 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.933 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.933 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.933 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:72 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:41.933 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.372400+0000", 2024-04-26T01:08:41.933 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 42.533895067000003, 2024-04-26T01:08:41.933 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018232822999999999, 2024-04-26T01:08:41.933 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.933 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.933 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:41.933 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:41.933 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:41.933 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:41.934 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 72 2024-04-26T01:08:41.934 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.934 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.934 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.934 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.934 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372400+0000", 2024-04-26T01:08:41.934 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.934 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.934 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.934 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.934 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372400+0000", 2024-04-26T01:08:41.934 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.934 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.935 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.935 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.935 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372401+0000", 2024-04-26T01:08:41.935 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7899999999999998e-07 2024-04-26T01:08:41.935 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.935 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.935 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.935 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372404+0000", 2024-04-26T01:08:41.935 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0469999999999998e-06 2024-04-26T01:08:41.935 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.936 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.936 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.936 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372413+0000", 2024-04-26T01:08:41.936 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3729999999999999e-06 2024-04-26T01:08:41.936 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.936 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.936 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.936 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.372419+0000", 2024-04-26T01:08:41.936 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1070000000000001e-06 2024-04-26T01:08:41.936 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.936 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.936 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.936 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374838+0000", 2024-04-26T01:08:41.936 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024188289999999999 2024-04-26T01:08:41.937 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.937 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.937 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:41.937 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374919+0000", 2024-04-26T01:08:41.937 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0717000000000007e-05 2024-04-26T01:08:41.937 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.937 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.937 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.937 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.381324+0000", 2024-04-26T01:08:41.937 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.006405666 2024-04-26T01:08:41.937 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.937 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.937 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:41.937 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.381402+0000", 2024-04-26T01:08:41.938 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7323000000000005e-05 2024-04-26T01:08:41.938 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.938 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.938 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.938 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.390633+0000", 2024-04-26T01:08:41.938 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0092311819999999992 2024-04-26T01:08:41.938 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.938 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.938 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.938 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.938 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.938 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpRead(3.cs2 33/30 ECSubRead(tid=19, to_read={3:3ca9ce7a:::smithi16428812-23:head=0,4096,0}, subchunks={3:3ca9ce7a:::smithi16428812-23:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:41.939 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.374332+0000", 2024-04-26T01:08:41.939 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 42.531962974000002, 2024-04-26T01:08:41.939 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.023990841999999998, 2024-04-26T01:08:41.939 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.939 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.939 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:41.939 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.939 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.939 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.939 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374332+0000", 2024-04-26T01:08:41.939 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.939 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.939 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.939 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.940 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374332+0000", 2024-04-26T01:08:41.940 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.940 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.940 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.940 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.940 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374333+0000", 2024-04-26T01:08:41.940 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0070000000000001e-06 2024-04-26T01:08:41.940 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.940 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.940 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.940 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374345+0000", 2024-04-26T01:08:41.940 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1853e-05 2024-04-26T01:08:41.940 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.940 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.941 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.941 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398055+0000", 2024-04-26T01:08:41.941 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.023710078999999998 2024-04-26T01:08:41.941 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.941 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.941 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.941 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398067+0000", 2024-04-26T01:08:41.941 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2455e-05 2024-04-26T01:08:41.941 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.941 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.941 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.941 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398160+0000", 2024-04-26T01:08:41.941 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2682000000000005e-05 2024-04-26T01:08:41.941 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.941 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.942 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.942 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398323+0000", 2024-04-26T01:08:41.942 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016276600000000001 2024-04-26T01:08:41.942 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.942 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.942 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.942 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.942 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.942 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:239 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~393216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:41.943 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.342338+0000", 2024-04-26T01:08:41.943 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 38.563957295000002, 2024-04-26T01:08:41.943 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.019617285000000002, 2024-04-26T01:08:41.943 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.943 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.943 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:41.943 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:41.943 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:41.943 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:41.943 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 239 2024-04-26T01:08:41.943 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.944 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.944 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.944 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.944 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342338+0000", 2024-04-26T01:08:41.944 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.944 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.944 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.944 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.944 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342338+0000", 2024-04-26T01:08:41.944 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.944 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.945 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.945 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.945 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342340+0000", 2024-04-26T01:08:41.945 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4700000000000001e-06 2024-04-26T01:08:41.945 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.945 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.945 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.945 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342351+0000", 2024-04-26T01:08:41.945 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0925e-05 2024-04-26T01:08:41.945 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.945 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.945 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.946 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342379+0000", 2024-04-26T01:08:41.946 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8140000000000002e-05 2024-04-26T01:08:41.946 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.946 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.946 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.946 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342395+0000", 2024-04-26T01:08:41.946 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5620999999999998e-05 2024-04-26T01:08:41.946 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.946 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.946 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.947 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342550+0000", 2024-04-26T01:08:41.947 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015483000000000001 2024-04-26T01:08:41.947 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.947 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.947 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op must wait for map", 2024-04-26T01:08:41.947 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342566+0000", 2024-04-26T01:08:41.947 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6214000000000001e-05 2024-04-26T01:08:41.947 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.947 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.947 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.947 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.349020+0000", 2024-04-26T01:08:41.947 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0064544240000000003 2024-04-26T01:08:41.947 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.947 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.948 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:41.948 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.349117+0000", 2024-04-26T01:08:41.948 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6212000000000007e-05 2024-04-26T01:08:41.948 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.948 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.948 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.948 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.361955+0000", 2024-04-26T01:08:41.948 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012838449 2024-04-26T01:08:41.948 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.948 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.948 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.949 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.949 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.949 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:240 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:41.949 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.342427+0000", 2024-04-26T01:08:41.949 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 38.563868085000003, 2024-04-26T01:08:41.949 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.019852153000000001, 2024-04-26T01:08:41.949 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.949 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.949 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:41.949 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:41.949 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:41.949 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:41.949 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 240 2024-04-26T01:08:41.949 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.949 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.950 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.950 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.950 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342427+0000", 2024-04-26T01:08:41.950 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.950 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.950 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.950 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.950 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342427+0000", 2024-04-26T01:08:41.950 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.950 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.950 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.950 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.950 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342428+0000", 2024-04-26T01:08:41.951 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2200000000000003e-07 2024-04-26T01:08:41.951 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.951 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.951 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.951 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342435+0000", 2024-04-26T01:08:41.951 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0180000000000004e-06 2024-04-26T01:08:41.951 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.951 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.951 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.951 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342451+0000", 2024-04-26T01:08:41.951 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6011e-05 2024-04-26T01:08:41.951 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.951 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.952 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.952 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342459+0000", 2024-04-26T01:08:41.952 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1370000000000002e-06 2024-04-26T01:08:41.952 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.952 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.952 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.952 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342766+0000", 2024-04-26T01:08:41.952 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00030684000000000001 2024-04-26T01:08:41.952 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.952 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.952 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:41.952 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342784+0000", 2024-04-26T01:08:41.952 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7773000000000001e-05 2024-04-26T01:08:41.952 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.953 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.953 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.953 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.349542+0000", 2024-04-26T01:08:41.953 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0067586349999999998 2024-04-26T01:08:41.953 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.953 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.953 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:41.953 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.349627+0000", 2024-04-26T01:08:41.953 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4790999999999995e-05 2024-04-26T01:08:41.953 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.953 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.953 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.953 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.362279+0000", 2024-04-26T01:08:41.953 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012652126 2024-04-26T01:08:41.954 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.954 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.954 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.954 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.954 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.954 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:241 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:41.954 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.342661+0000", 2024-04-26T01:08:41.954 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 38.563634235000002, 2024-04-26T01:08:41.954 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.020164171000000002, 2024-04-26T01:08:41.954 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.954 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.954 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:41.954 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:41.954 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:41.955 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:41.955 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 241 2024-04-26T01:08:41.955 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.955 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.955 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.955 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.955 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342661+0000", 2024-04-26T01:08:41.955 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.955 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.955 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.955 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.955 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342661+0000", 2024-04-26T01:08:41.955 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.955 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.956 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.956 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.956 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342662+0000", 2024-04-26T01:08:41.956 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0100000000000003e-07 2024-04-26T01:08:41.956 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.956 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.956 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.956 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342668+0000", 2024-04-26T01:08:41.956 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2369999999999997e-06 2024-04-26T01:08:41.957 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.957 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.957 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.957 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342682+0000", 2024-04-26T01:08:41.957 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4166e-05 2024-04-26T01:08:41.957 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.957 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.957 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.957 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342691+0000", 2024-04-26T01:08:41.957 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8000000000000004e-06 2024-04-26T01:08:41.957 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.957 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.958 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.958 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342844+0000", 2024-04-26T01:08:41.958 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015316599999999999 2024-04-26T01:08:41.958 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.958 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.958 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:41.958 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342853+0000", 2024-04-26T01:08:41.958 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9700000000000005e-06 2024-04-26T01:08:41.958 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.958 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.958 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.958 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.350630+0000", 2024-04-26T01:08:41.959 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0077770890000000001 2024-04-26T01:08:41.959 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.959 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.959 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:41.959 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.350718+0000", 2024-04-26T01:08:41.959 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8156000000000001e-05 2024-04-26T01:08:41.959 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.959 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.959 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.959 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.362825+0000", 2024-04-26T01:08:41.959 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012106686 2024-04-26T01:08:41.959 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.959 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.959 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.960 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.960 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.960 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:276 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:41.960 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.413967+0000", 2024-04-26T01:08:41.960 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 37.492327619999998, 2024-04-26T01:08:41.960 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014303402, 2024-04-26T01:08:41.960 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.960 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.960 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:41.960 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:41.960 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:41.960 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:41.961 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 276 2024-04-26T01:08:41.961 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.961 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.961 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.961 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.961 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413967+0000", 2024-04-26T01:08:41.961 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.961 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.961 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.961 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.961 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413967+0000", 2024-04-26T01:08:41.961 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.961 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.961 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.962 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.962 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413969+0000", 2024-04-26T01:08:41.962 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.24e-06 2024-04-26T01:08:41.962 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.962 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.962 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.962 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.413996+0000", 2024-04-26T01:08:41.962 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7093999999999998e-05 2024-04-26T01:08:41.962 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.963 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.963 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.963 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414016+0000", 2024-04-26T01:08:41.963 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0072999999999999e-05 2024-04-26T01:08:41.963 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.963 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.964 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.964 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.414031+0000", 2024-04-26T01:08:41.964 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5159e-05 2024-04-26T01:08:41.964 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.964 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.964 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.964 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.420581+0000", 2024-04-26T01:08:41.964 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0065497519999999998 2024-04-26T01:08:41.964 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.964 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.964 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:41.964 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.420691+0000", 2024-04-26T01:08:41.964 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000110259 2024-04-26T01:08:41.964 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.965 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.965 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.965 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.428271+0000", 2024-04-26T01:08:41.965 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0075798250000000001 2024-04-26T01:08:41.965 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.965 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.965 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.965 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.965 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.965 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs2 41/30 ECSubWrite(tid=91, reqid=client.4521.0:345, at_version=41'27, trim_to=0'0, roll_forward_to=41'26))", 2024-04-26T01:08:41.965 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:07.610935+0000", 2024-04-26T01:08:41.965 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 34.295359712, 2024-04-26T01:08:41.965 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.629675231, 2024-04-26T01:08:41.965 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.965 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.966 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:41.966 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.966 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.966 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.966 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.610935+0000", 2024-04-26T01:08:41.966 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.966 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.966 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.966 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.966 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.610935+0000", 2024-04-26T01:08:41.966 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.966 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.966 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.967 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.967 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.610938+0000", 2024-04-26T01:08:41.967 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.39e-06 2024-04-26T01:08:41.967 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.967 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.967 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.967 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.611405+0000", 2024-04-26T01:08:41.967 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00046744899999999999 2024-04-26T01:08:41.967 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.967 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.967 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.968 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238064+0000", 2024-04-26T01:08:41.968 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.62665866100000001 2024-04-26T01:08:41.968 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.968 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.968 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.968 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238080+0000", 2024-04-26T01:08:41.968 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6623e-05 2024-04-26T01:08:41.968 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.968 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.968 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.968 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238214+0000", 2024-04-26T01:08:41.968 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013316900000000001 2024-04-26T01:08:41.968 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.968 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.969 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:41.969 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238235+0000", 2024-04-26T01:08:41.969 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0937e-05 2024-04-26T01:08:41.969 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.969 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.969 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:41.969 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.240510+0000", 2024-04-26T01:08:41.969 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022752860000000001 2024-04-26T01:08:41.969 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.969 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.969 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.969 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.240611+0000", 2024-04-26T01:08:41.969 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000100716 2024-04-26T01:08:41.969 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.969 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.970 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.970 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.970 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.970 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 43/30 ECSubWriteReply(tid=33, last_complete=43'17, committed=1, applied=1))", 2024-04-26T01:08:41.970 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.473084+0000", 2024-04-26T01:08:41.970 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 32.433211469, 2024-04-26T01:08:41.970 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.913281966, 2024-04-26T01:08:41.970 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.971 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.971 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:41.971 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.971 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.971 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.971 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.473084+0000", 2024-04-26T01:08:41.971 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.971 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.971 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.971 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.971 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.473084+0000", 2024-04-26T01:08:41.971 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.972 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.972 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.972 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.972 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.473084+0000", 2024-04-26T01:08:41.972 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6300000000000005e-07 2024-04-26T01:08:41.972 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.972 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.972 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.972 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.473088+0000", 2024-04-26T01:08:41.972 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7620000000000001e-06 2024-04-26T01:08:41.972 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.972 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.972 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.973 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.386034+0000", 2024-04-26T01:08:41.973 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.91294643399999997 2024-04-26T01:08:41.973 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.973 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.973 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.973 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.386055+0000", 2024-04-26T01:08:41.973 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0492999999999998e-05 2024-04-26T01:08:41.973 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.973 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.973 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.973 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.386234+0000", 2024-04-26T01:08:41.973 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000179259 2024-04-26T01:08:41.973 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.973 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.974 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.974 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.386366+0000", 2024-04-26T01:08:41.974 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013135499999999999 2024-04-26T01:08:41.974 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.974 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.974 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.974 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.974 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.974 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:593 3.6s0 3:69323dd0:::smithi16428812-38:head [read 0~1884160,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:41.974 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.164747+0000", 2024-04-26T01:08:41.974 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.741548418000001, 2024-04-26T01:08:41.975 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014725004, 2024-04-26T01:08:41.975 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.975 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.975 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:41.975 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:41.975 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:41.975 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:41.975 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 593 2024-04-26T01:08:41.975 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.975 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.975 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.975 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.975 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164747+0000", 2024-04-26T01:08:41.975 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.976 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.976 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.976 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.976 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164747+0000", 2024-04-26T01:08:41.976 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.976 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.976 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.976 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.976 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164748+0000", 2024-04-26T01:08:41.976 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0410000000000001e-06 2024-04-26T01:08:41.976 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.976 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.976 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.976 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164754+0000", 2024-04-26T01:08:41.976 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3929999999999996e-06 2024-04-26T01:08:41.977 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.977 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.977 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.977 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164769+0000", 2024-04-26T01:08:41.977 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4834999999999999e-05 2024-04-26T01:08:41.977 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.977 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.977 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.977 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164779+0000", 2024-04-26T01:08:41.978 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0117e-05 2024-04-26T01:08:41.978 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.978 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.978 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.978 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.165879+0000", 2024-04-26T01:08:41.978 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0010995899999999999 2024-04-26T01:08:41.978 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.978 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.978 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:41.978 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.166031+0000", 2024-04-26T01:08:41.979 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015242399999999999 2024-04-26T01:08:41.979 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.979 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.979 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.979 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.179472+0000", 2024-04-26T01:08:41.979 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013440604 2024-04-26T01:08:41.979 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.980 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.980 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.980 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.980 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.980 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:594 3.6s0 3:69323dd0:::smithi16428812-38:head [read 0~1884160,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:41.980 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.164820+0000", 2024-04-26T01:08:41.980 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.741474844999999, 2024-04-26T01:08:41.980 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.016408565, 2024-04-26T01:08:41.980 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.980 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.980 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:41.980 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:41.981 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:41.981 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:41.981 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 594 2024-04-26T01:08:41.981 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.981 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.981 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.981 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.981 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164820+0000", 2024-04-26T01:08:41.981 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.981 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.981 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.981 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.981 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164820+0000", 2024-04-26T01:08:41.981 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.981 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.982 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.982 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.982 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164821+0000", 2024-04-26T01:08:41.982 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2099999999999995e-07 2024-04-26T01:08:41.982 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.982 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.982 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.982 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164826+0000", 2024-04-26T01:08:41.982 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4430000000000002e-06 2024-04-26T01:08:41.982 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.982 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.982 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.982 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164839+0000", 2024-04-26T01:08:41.983 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2622999999999999e-05 2024-04-26T01:08:41.983 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.983 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.983 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.983 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.164851+0000", 2024-04-26T01:08:41.983 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1497e-05 2024-04-26T01:08:41.983 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.983 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.983 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.983 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.166481+0000", 2024-04-26T01:08:41.983 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001630432 2024-04-26T01:08:41.983 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.983 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.983 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:41.984 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.166572+0000", 2024-04-26T01:08:41.984 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1268999999999996e-05 2024-04-26T01:08:41.984 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.984 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.984 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.984 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.181229+0000", 2024-04-26T01:08:41.984 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.01465648 2024-04-26T01:08:41.984 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.984 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.984 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.984 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.984 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.985 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/30 ECSubWrite(tid=318, reqid=client.4521.0:601, at_version=58'41, trim_to=0'0, roll_forward_to=58'39))", 2024-04-26T01:08:41.985 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.359815+0000", 2024-04-26T01:08:41.985 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.546480424999999, 2024-04-26T01:08:41.985 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.11192218399999999, 2024-04-26T01:08:41.985 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.985 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.985 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:41.985 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.985 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.985 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.985 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.359815+0000", 2024-04-26T01:08:41.985 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.985 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.985 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.985 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.986 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.359815+0000", 2024-04-26T01:08:41.986 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.986 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.986 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.986 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.986 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.359816+0000", 2024-04-26T01:08:41.986 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1429999999999999e-06 2024-04-26T01:08:41.986 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.986 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.986 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.986 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.359875+0000", 2024-04-26T01:08:41.986 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9364000000000003e-05 2024-04-26T01:08:41.986 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.986 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.987 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.987 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.470037+0000", 2024-04-26T01:08:41.987 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.110162076 2024-04-26T01:08:41.987 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.987 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.987 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.987 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.470055+0000", 2024-04-26T01:08:41.987 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7863000000000002e-05 2024-04-26T01:08:41.987 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.987 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.987 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.987 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.470147+0000", 2024-04-26T01:08:41.987 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2103999999999998e-05 2024-04-26T01:08:41.987 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.987 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.988 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:41.988 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.470157+0000", 2024-04-26T01:08:41.988 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0182e-05 2024-04-26T01:08:41.988 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.988 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.988 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:41.988 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.471667+0000", 2024-04-26T01:08:41.988 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015096739999999999 2024-04-26T01:08:41.988 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.989 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.989 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.989 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.471737+0000", 2024-04-26T01:08:41.989 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9777999999999995e-05 2024-04-26T01:08:41.989 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.989 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.989 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.989 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.989 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.989 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs2 59/44 ECSubWrite(tid=46, reqid=client.4521.0:635, at_version=59'56, trim_to=0'0, roll_forward_to=59'54))", 2024-04-26T01:08:41.989 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:25.524509+0000", 2024-04-26T01:08:41.989 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 16.381786171999998, 2024-04-26T01:08:41.989 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24557240299999999, 2024-04-26T01:08:41.989 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.524509+0000", 2024-04-26T01:08:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.524509+0000", 2024-04-26T01:08:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.524510+0000", 2024-04-26T01:08:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4199999999999995e-07 2024-04-26T01:08:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.524523+0000", 2024-04-26T01:08:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3491e-05 2024-04-26T01:08:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.768874+0000", 2024-04-26T01:08:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24435099299999999 2024-04-26T01:08:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.768892+0000", 2024-04-26T01:08:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7886000000000001e-05 2024-04-26T01:08:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.768997+0000", 2024-04-26T01:08:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010476000000000001 2024-04-26T01:08:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.769012+0000", 2024-04-26T01:08:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5641000000000002e-05 2024-04-26T01:08:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.770034+0000", 2024-04-26T01:08:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001022071 2024-04-26T01:08:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.770081+0000", 2024-04-26T01:08:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6919000000000003e-05 2024-04-26T01:08:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:641 3.6s0 3:69323dd0:::smithi16428812-38:head [delete] snapc f=[f] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e60)", 2024-04-26T01:08:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:26.167403+0000", 2024-04-26T01:08:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 15.738892047, 2024-04-26T01:08:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.025490321999999999, 2024-04-26T01:08:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 641 2024-04-26T01:08:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167403+0000", 2024-04-26T01:08:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167403+0000", 2024-04-26T01:08:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167406+0000", 2024-04-26T01:08:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7920000000000001e-06 2024-04-26T01:08:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167415+0000", 2024-04-26T01:08:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1540000000000008e-06 2024-04-26T01:08:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167442+0000", 2024-04-26T01:08:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6806999999999999e-05 2024-04-26T01:08:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167457+0000", 2024-04-26T01:08:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4836e-05 2024-04-26T01:08:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167580+0000", 2024-04-26T01:08:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012298800000000001 2024-04-26T01:08:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op must wait for map", 2024-04-26T01:08:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167594+0000", 2024-04-26T01:08:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4328000000000001e-05 2024-04-26T01:08:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.172291+0000", 2024-04-26T01:08:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046972139999999999 2024-04-26T01:08:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.172409+0000", 2024-04-26T01:08:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000118066 2024-04-26T01:08:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.173600+0000", 2024-04-26T01:08:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001190839 2024-04-26T01:08:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.180944+0000", 2024-04-26T01:08:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0073435790000000003 2024-04-26T01:08:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.192802+0000", 2024-04-26T01:08:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011858848999999999 2024-04-26T01:08:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.192893+0000", 2024-04-26T01:08:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0870000000000002e-05 2024-04-26T01:08:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpRead(3.7s3 63/44 ECSubRead(tid=373, to_read={3:e163c084:::smithi16428812-22:head=0,491520,0}, subchunks={3:e163c084:::smithi16428812-22:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:29.210385+0000", 2024-04-26T01:08:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.695909866000001, 2024-04-26T01:08:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.38767122700000001, 2024-04-26T01:08:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.210385+0000", 2024-04-26T01:08:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.210385+0000", 2024-04-26T01:08:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.210386+0000", 2024-04-26T01:08:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3399999999999999e-07 2024-04-26T01:08:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.210398+0000", 2024-04-26T01:08:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2553e-05 2024-04-26T01:08:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.597243+0000", 2024-04-26T01:08:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.38684432099999999 2024-04-26T01:08:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.597261+0000", 2024-04-26T01:08:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7861999999999999e-05 2024-04-26T01:08:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.597490+0000", 2024-04-26T01:08:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000228982 2024-04-26T01:08:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.598056+0000", 2024-04-26T01:08:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000566875 2024-04-26T01:08:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpRead(3.7s3 63/44 ECSubRead(tid=382, to_read={3:ed7aa414:::smithi16428812-13:head=0,53248,0}, subchunks={3:ed7aa414:::smithi16428812-13:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:29.786784+0000", 2024-04-26T01:08:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.119511320999999, 2024-04-26T01:08:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.36638038000000001, 2024-04-26T01:08:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.786784+0000", 2024-04-26T01:08:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.786784+0000", 2024-04-26T01:08:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.786786+0000", 2024-04-26T01:08:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0269999999999998e-06 2024-04-26T01:08:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.786810+0000", 2024-04-26T01:08:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4260000000000002e-05 2024-04-26T01:08:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:30.152620+0000", 2024-04-26T01:08:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.36580987500000001 2024-04-26T01:08:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:30.152649+0000", 2024-04-26T01:08:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.864e-05 2024-04-26T01:08:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:30.152857+0000", 2024-04-26T01:08:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00020862800000000001 2024-04-26T01:08:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:30.153164+0000", 2024-04-26T01:08:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00030695000000000001 2024-04-26T01:08:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:743 3.6s0 3:6387e095:::smithi16428812-24:head [append 0~606208] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:08:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.167557+0000", 2024-04-26T01:08:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.7387383769999998, 2024-04-26T01:08:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014902627, 2024-04-26T01:08:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 743 2024-04-26T01:08:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.167557+0000", 2024-04-26T01:08:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.167557+0000", 2024-04-26T01:08:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.167559+0000", 2024-04-26T01:08:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3199999999999998e-06 2024-04-26T01:08:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.168345+0000", 2024-04-26T01:08:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00078589199999999995 2024-04-26T01:08:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.168372+0000", 2024-04-26T01:08:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7063000000000002e-05 2024-04-26T01:08:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.168383+0000", 2024-04-26T01:08:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1457999999999999e-05 2024-04-26T01:08:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.168478+0000", 2024-04-26T01:08:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4920999999999995e-05 2024-04-26T01:08:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.168602+0000", 2024-04-26T01:08:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012403499999999999 2024-04-26T01:08:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.169780+0000", 2024-04-26T01:08:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0011779169999999999 2024-04-26T01:08:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.174155+0000", 2024-04-26T01:08:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0043748099999999998 2024-04-26T01:08:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.182360+0000", 2024-04-26T01:08:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0082048520000000003 2024-04-26T01:08:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.182459+0000", 2024-04-26T01:08:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9358999999999995e-05 2024-04-26T01:08:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.016 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:42.018 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:08:42.019 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:08:42.019 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:08:42.019 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:42.020 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:42.020 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:42.020 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:42.020 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:41.988+0000 7f4b5ffff640 0 -- 172.21.15.8:0/1897956200 >> [v2:172.21.15.116:6800/2158650188,v1:172.21.15.116:6801/2158650188] conn(0x7f4b6021d8f0 msgr2=0x7f4b6021fce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:08:42.021 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:42.016+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:42.021 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:42.022 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:42.043 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.12 to 1 (10000) 2024-04-26T01:08:42.043 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-04-26T01:08:42.047 INFO:tasks.rados.rados.0.smithi164.stdout:329: done (0 left) 2024-04-26T01:08:42.047 INFO:tasks.rados.rados.0.smithi164.stdout:330: setattr oid 45 current snap is 22 2024-04-26T01:08:42.053 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 4 current snap is 22 2024-04-26T01:08:42.053 INFO:tasks.rados.rados.0.smithi164.stdout:331: seq_num 144 ranges {2719744=16384} 2024-04-26T01:08:42.054 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:42.054 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:42.078 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-04-26T01:08:42.098 INFO:tasks.rados.rados.0.smithi164.stdout:331: writing smithi16428812-4 from 2719744 to 2736128 tid 1 2024-04-26T01:08:42.098 INFO:tasks.rados.rados.0.smithi164.stdout:332: delete oid 7 current snap is 22 2024-04-26T01:08:42.098 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 45 v 101 (ObjNum 177 snap 0 seq_num 1) dirty exists 2024-04-26T01:08:42.106 INFO:tasks.rados.rados.0.smithi164.stdout:331: finishing write tid 1 to smithi16428812-4 2024-04-26T01:08:42.109 INFO:tasks.rados.rados.0.smithi164.stdout:331: finishing write tid 2 to smithi16428812-4 2024-04-26T01:08:42.109 INFO:tasks.rados.rados.0.smithi164.stdout:331: finishing write tid 3 to smithi16428812-4 2024-04-26T01:08:42.109 INFO:tasks.rados.rados.0.smithi164.stdout:331: oid 4 updating version 0 to 104 2024-04-26T01:08:42.109 INFO:tasks.rados.rados.0.smithi164.stdout:331: oid 4 version 104 is already newer than 103 2024-04-26T01:08:42.109 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 4 v 104 (ObjNum 144 snap 22 seq_num 144) dirty exists 2024-04-26T01:08:42.110 INFO:tasks.rados.rados.0.smithi164.stdout:331: left oid 4 (ObjNum 144 snap 22 seq_num 144) 2024-04-26T01:08:42.119 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:42.119+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:42.158 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:42.158 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:42.158 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-04-26T01:08:42.158 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:42.173 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:08:42.219 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:42.217+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:42.320 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:42.317+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:42.335 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:42.335 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:42.336 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-04-26T01:08:42.336 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-04-26T01:08:42.336 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:42.349 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:08:42.422 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:42.420+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:42.476 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:42.477 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:42.477 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:42.477 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:42.477 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.477 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:65 3.7s0 3:e163c084:::smithi16428812-22:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:42.477 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.364410+0000", 2024-04-26T01:08:42.477 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 43.109614649999997, 2024-04-26T01:08:42.477 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.081696361999999995, 2024-04-26T01:08:42.477 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.477 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.477 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:42.477 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:42.477 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:42.478 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:42.478 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 65 2024-04-26T01:08:42.478 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.478 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.478 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.478 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.478 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364410+0000", 2024-04-26T01:08:42.478 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.478 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.478 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.478 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.478 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364410+0000", 2024-04-26T01:08:42.478 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.478 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.479 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.479 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.479 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364411+0000", 2024-04-26T01:08:42.479 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.102e-06 2024-04-26T01:08:42.479 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.479 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.479 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.479 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364416+0000", 2024-04-26T01:08:42.479 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8389999999999997e-06 2024-04-26T01:08:42.479 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.479 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.479 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.479 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364424+0000", 2024-04-26T01:08:42.479 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6100000000000006e-06 2024-04-26T01:08:42.480 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.480 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.480 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.480 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364432+0000", 2024-04-26T01:08:42.480 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5320000000000001e-06 2024-04-26T01:08:42.480 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.480 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.480 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.480 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.365925+0000", 2024-04-26T01:08:42.480 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001493592 2024-04-26T01:08:42.480 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.480 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.480 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:42.480 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366035+0000", 2024-04-26T01:08:42.481 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000109448 2024-04-26T01:08:42.481 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.481 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.481 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:42.481 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366490+0000", 2024-04-26T01:08:42.481 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000455427 2024-04-26T01:08:42.481 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.481 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.481 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:42.481 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.367513+0000", 2024-04-26T01:08:42.481 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001023014 2024-04-26T01:08:42.481 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.481 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.481 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:42.481 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446008+0000", 2024-04-26T01:08:42.482 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.078494871999999993 2024-04-26T01:08:42.482 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.482 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.482 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.482 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446106+0000", 2024-04-26T01:08:42.482 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7925999999999997e-05 2024-04-26T01:08:42.482 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.482 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.482 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.482 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.482 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.482 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:66 3.7s0 3:e163c084:::smithi16428812-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:42.482 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.364442+0000", 2024-04-26T01:08:42.483 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 43.109582121999999, 2024-04-26T01:08:42.483 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.088465429999999998, 2024-04-26T01:08:42.483 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.483 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.483 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:42.483 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:42.483 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:42.483 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:42.483 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 66 2024-04-26T01:08:42.483 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.483 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.483 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.483 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.483 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364442+0000", 2024-04-26T01:08:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364442+0000", 2024-04-26T01:08:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364443+0000", 2024-04-26T01:08:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7899999999999998e-07 2024-04-26T01:08:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364446+0000", 2024-04-26T01:08:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9119999999999998e-06 2024-04-26T01:08:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364454+0000", 2024-04-26T01:08:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0229999999999992e-06 2024-04-26T01:08:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.364461+0000", 2024-04-26T01:08:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0110000000000003e-06 2024-04-26T01:08:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.367245+0000", 2024-04-26T01:08:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027847499999999999 2024-04-26T01:08:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.367332+0000", 2024-04-26T01:08:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6118000000000005e-05 2024-04-26T01:08:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446632+0000", 2024-04-26T01:08:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.079300689999999993 2024-04-26T01:08:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:42.487 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.446716+0000", 2024-04-26T01:08:42.487 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3391000000000002e-05 2024-04-26T01:08:42.487 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.487 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.487 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.487 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.452908+0000", 2024-04-26T01:08:42.487 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0061919560000000002 2024-04-26T01:08:42.488 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.488 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.488 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.488 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.488 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.488 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.5s2 37/30 ECSubWrite(tid=22, reqid=client.4521.0:251, at_version=37'17, trim_to=0'0, roll_forward_to=36'13))", 2024-04-26T01:08:42.488 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.434740+0000", 2024-04-26T01:08:42.488 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 39.039284133000002, 2024-04-26T01:08:42.488 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.47257335099999997, 2024-04-26T01:08:42.488 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.489 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.489 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:42.489 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.489 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.489 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.489 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434740+0000", 2024-04-26T01:08:42.489 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.489 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.489 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.489 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.489 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434740+0000", 2024-04-26T01:08:42.489 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.489 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.489 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.490 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.490 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434741+0000", 2024-04-26T01:08:42.490 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8800000000000002e-07 2024-04-26T01:08:42.490 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.490 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.490 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.490 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434764+0000", 2024-04-26T01:08:42.490 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3499999999999999e-05 2024-04-26T01:08:42.490 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.490 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.490 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.490 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906023+0000", 2024-04-26T01:08:42.490 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.47125867799999999 2024-04-26T01:08:42.490 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.490 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.491 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.491 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906041+0000", 2024-04-26T01:08:42.491 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7887e-05 2024-04-26T01:08:42.491 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.491 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.491 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.491 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906172+0000", 2024-04-26T01:08:42.491 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013065200000000001 2024-04-26T01:08:42.491 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.491 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.491 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:42.491 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906192+0000", 2024-04-26T01:08:42.491 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0363000000000001e-05 2024-04-26T01:08:42.491 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.492 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.492 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:42.493 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.907240+0000", 2024-04-26T01:08:42.493 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0010479 2024-04-26T01:08:42.493 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.493 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.493 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.493 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.907314+0000", 2024-04-26T01:08:42.494 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3583000000000003e-05 2024-04-26T01:08:42.494 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.494 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.494 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.494 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.494 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.494 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 50/44 ECSubWriteReply(tid=226, last_complete=50'42, committed=1, applied=1))", 2024-04-26T01:08:42.494 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:16.093834+0000", 2024-04-26T01:08:42.494 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 26.380190697, 2024-04-26T01:08:42.494 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.069647735000000002, 2024-04-26T01:08:42.494 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.495 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.495 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:42.495 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.495 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.495 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.495 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.093834+0000", 2024-04-26T01:08:42.495 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.495 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.495 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.495 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.093834+0000", 2024-04-26T01:08:42.495 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.495 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.093834+0000", 2024-04-26T01:08:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9800000000000003e-07 2024-04-26T01:08:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.093838+0000", 2024-04-26T01:08:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8369999999999999e-06 2024-04-26T01:08:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.163332+0000", 2024-04-26T01:08:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.069493965000000005 2024-04-26T01:08:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.497 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.497 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.163346+0000", 2024-04-26T01:08:42.497 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.341e-05 2024-04-26T01:08:42.497 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.497 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.497 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.497 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.163447+0000", 2024-04-26T01:08:42.497 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000101859 2024-04-26T01:08:42.497 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.497 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.497 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.497 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.163481+0000", 2024-04-26T01:08:42.497 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3966e-05 2024-04-26T01:08:42.497 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.498 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.498 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.498 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.498 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.498 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 52/30 ECSubWriteReply(tid=246, last_complete=52'26, committed=1, applied=1))", 2024-04-26T01:08:42.498 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:18.467431+0000", 2024-04-26T01:08:42.498 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 24.006593210999998, 2024-04-26T01:08:42.498 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.59825841400000002, 2024-04-26T01:08:42.498 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.498 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.498 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:42.498 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.498 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.498 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.499 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:18.467431+0000", 2024-04-26T01:08:42.499 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.499 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.499 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.499 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.499 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:18.467431+0000", 2024-04-26T01:08:42.499 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.499 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.499 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.499 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.499 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:18.467432+0000", 2024-04-26T01:08:42.499 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5400000000000001e-07 2024-04-26T01:08:42.499 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.499 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.500 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.500 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:18.467437+0000", 2024-04-26T01:08:42.500 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1200000000000001e-06 2024-04-26T01:08:42.500 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.500 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.500 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.500 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.065059+0000", 2024-04-26T01:08:42.500 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.59762205800000001 2024-04-26T01:08:42.500 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.500 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.500 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.500 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.065068+0000", 2024-04-26T01:08:42.500 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9430000000000006e-06 2024-04-26T01:08:42.500 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.500 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.501 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.501 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.065136+0000", 2024-04-26T01:08:42.501 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7841000000000003e-05 2024-04-26T01:08:42.501 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.501 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.501 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.501 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.065690+0000", 2024-04-26T01:08:42.501 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00055379799999999999 2024-04-26T01:08:42.501 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.501 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.501 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.501 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.501 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.501 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 52/30 ECSubWriteReply(tid=247, last_complete=52'26, committed=1, applied=1))", 2024-04-26T01:08:42.502 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:19.067196+0000", 2024-04-26T01:08:42.502 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 23.406828256000001, 2024-04-26T01:08:42.502 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.12773290200000001, 2024-04-26T01:08:42.502 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.502 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.502 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:42.502 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.502 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.502 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.502 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.067196+0000", 2024-04-26T01:08:42.502 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.502 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.502 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.502 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.503 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.067196+0000", 2024-04-26T01:08:42.503 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.503 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.503 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.503 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.503 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.067197+0000", 2024-04-26T01:08:42.503 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1699999999999997e-07 2024-04-26T01:08:42.503 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.503 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.503 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.503 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.067201+0000", 2024-04-26T01:08:42.503 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5340000000000001e-06 2024-04-26T01:08:42.503 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.503 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.504 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.504 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.194638+0000", 2024-04-26T01:08:42.504 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.12743633600000001 2024-04-26T01:08:42.504 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.504 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.504 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.504 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.194653+0000", 2024-04-26T01:08:42.504 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5194000000000001e-05 2024-04-26T01:08:42.504 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.504 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.504 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.504 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.194810+0000", 2024-04-26T01:08:42.504 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015676500000000001 2024-04-26T01:08:42.504 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.504 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.505 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.505 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.194929+0000", 2024-04-26T01:08:42.505 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000119356 2024-04-26T01:08:42.505 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.505 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.505 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.505 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.505 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.505 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:599 3.fs0 3:fa21e776:::smithi16428812-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:42.505 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.187146+0000", 2024-04-26T01:08:42.505 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 18.286878077000001, 2024-04-26T01:08:42.505 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.094712416999999993, 2024-04-26T01:08:42.505 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.505 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.505 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:42.506 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:42.506 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:42.506 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:42.506 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 599 2024-04-26T01:08:42.506 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.506 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.506 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.506 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.506 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.187146+0000", 2024-04-26T01:08:42.506 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.506 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.506 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.506 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.507 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.187146+0000", 2024-04-26T01:08:42.507 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.507 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.507 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.507 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.507 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.187147+0000", 2024-04-26T01:08:42.507 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7800000000000001e-07 2024-04-26T01:08:42.507 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.507 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.507 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.507 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.187151+0000", 2024-04-26T01:08:42.508 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.546e-06 2024-04-26T01:08:42.508 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.508 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.508 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.508 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.187161+0000", 2024-04-26T01:08:42.508 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0304e-05 2024-04-26T01:08:42.508 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.508 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.508 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.508 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.187166+0000", 2024-04-26T01:08:42.508 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2429999999999996e-06 2024-04-26T01:08:42.508 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.508 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.509 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.509 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.205675+0000", 2024-04-26T01:08:42.509 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018509105000000001 2024-04-26T01:08:42.509 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.509 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.509 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:42.509 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.205807+0000", 2024-04-26T01:08:42.509 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013163899999999999 2024-04-26T01:08:42.509 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.509 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.509 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.509 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.212468+0000", 2024-04-26T01:08:42.509 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0066608329999999997 2024-04-26T01:08:42.509 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.509 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.510 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:42.510 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.212545+0000", 2024-04-26T01:08:42.510 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7768999999999994e-05 2024-04-26T01:08:42.510 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.510 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.510 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.510 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.281859+0000", 2024-04-26T01:08:42.510 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.069313200000000005 2024-04-26T01:08:42.510 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.510 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.510 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.510 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.510 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.511 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 58/44 ECSubWriteReply(tid=310, last_complete=58'73, committed=1, applied=1))", 2024-04-26T01:08:42.511 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.213740+0000", 2024-04-26T01:08:42.511 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 18.260283932, 2024-04-26T01:08:42.511 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.066820729999999995, 2024-04-26T01:08:42.511 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.511 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.511 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:42.511 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.511 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.511 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.511 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.213740+0000", 2024-04-26T01:08:42.511 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.511 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.511 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.512 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.512 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.213740+0000", 2024-04-26T01:08:42.512 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.512 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.512 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.512 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.512 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.213741+0000", 2024-04-26T01:08:42.512 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9299999999999999e-07 2024-04-26T01:08:42.512 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.512 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.512 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.512 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.213742+0000", 2024-04-26T01:08:42.512 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6130000000000001e-06 2024-04-26T01:08:42.512 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.512 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.513 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.513 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.279951+0000", 2024-04-26T01:08:42.513 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.066209009999999999 2024-04-26T01:08:42.513 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.513 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.513 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.513 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.279960+0000", 2024-04-26T01:08:42.513 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8440000000000004e-06 2024-04-26T01:08:42.513 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.513 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.513 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.513 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.280027+0000", 2024-04-26T01:08:42.513 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6778000000000003e-05 2024-04-26T01:08:42.513 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.514 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.514 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.514 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.280561+0000", 2024-04-26T01:08:42.514 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00053429200000000001 2024-04-26T01:08:42.514 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.514 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.514 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.514 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.514 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.514 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:601 3.bs0 3:d5691d7a:::smithi16428812-40:head [copy-from ver 4] snapc f=[f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:42.514 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.340640+0000", 2024-04-26T01:08:42.514 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 18.133384349, 2024-04-26T01:08:42.515 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.78325740899999996, 2024-04-26T01:08:42.515 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.515 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.515 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:42.515 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:42.515 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:42.515 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:42.515 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 601 2024-04-26T01:08:42.515 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.515 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.516 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.516 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.516 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340640+0000", 2024-04-26T01:08:42.516 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.516 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.516 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.516 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.516 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340640+0000", 2024-04-26T01:08:42.516 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.516 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.516 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.516 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.516 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340644+0000", 2024-04-26T01:08:42.516 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9169999999999999e-06 2024-04-26T01:08:42.516 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.517 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.517 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.517 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340661+0000", 2024-04-26T01:08:42.517 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7390000000000001e-05 2024-04-26T01:08:42.517 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.517 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.517 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.517 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340696+0000", 2024-04-26T01:08:42.517 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4332999999999998e-05 2024-04-26T01:08:42.517 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.517 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.517 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.517 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340728+0000", 2024-04-26T01:08:42.517 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.2246000000000003e-05 2024-04-26T01:08:42.518 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.518 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.518 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.518 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340944+0000", 2024-04-26T01:08:42.518 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00021597700000000001 2024-04-26T01:08:42.518 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.518 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.518 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:42.518 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341149+0000", 2024-04-26T01:08:42.518 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000204725 2024-04-26T01:08:42.518 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.518 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.518 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:42.519 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.360012+0000", 2024-04-26T01:08:42.519 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018863683999999999 2024-04-26T01:08:42.519 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.519 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.519 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:42.519 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.362478+0000", 2024-04-26T01:08:42.519 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024654849999999999 2024-04-26T01:08:42.519 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.519 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.519 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:42.519 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.123741+0000", 2024-04-26T01:08:42.519 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.76126282099999998 2024-04-26T01:08:42.519 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.519 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.519 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.520 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.123897+0000", 2024-04-26T01:08:42.520 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015683099999999999 2024-04-26T01:08:42.520 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.520 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.520 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.520 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.520 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.520 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 62/44 ECSubWriteReply(tid=370, last_complete=63'73, committed=1, applied=1))", 2024-04-26T01:08:42.520 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:29.200166+0000", 2024-04-26T01:08:42.520 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 13.273858386000001, 2024-04-26T01:08:42.520 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.42031032000000002, 2024-04-26T01:08:42.520 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.520 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.520 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:42.520 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.521 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.521 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.521 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.200166+0000", 2024-04-26T01:08:42.521 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.521 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.521 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.521 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.521 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.200166+0000", 2024-04-26T01:08:42.521 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.521 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.521 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.521 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.521 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.200168+0000", 2024-04-26T01:08:42.521 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0370000000000001e-06 2024-04-26T01:08:42.522 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.522 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.522 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.522 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.200180+0000", 2024-04-26T01:08:42.522 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1633e-05 2024-04-26T01:08:42.522 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.522 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.522 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.522 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.619636+0000", 2024-04-26T01:08:42.522 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.41945664399999999 2024-04-26T01:08:42.522 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.522 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.522 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.522 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.619652+0000", 2024-04-26T01:08:42.523 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5673000000000001e-05 2024-04-26T01:08:42.523 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.523 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.523 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.523 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.619741+0000", 2024-04-26T01:08:42.523 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9479000000000002e-05 2024-04-26T01:08:42.523 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.523 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.523 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.523 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.620476+0000", 2024-04-26T01:08:42.523 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00073485399999999995 2024-04-26T01:08:42.524 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.524 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.524 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.524 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.524 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.524 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:688 3.7s0 3:ed7aa414:::smithi16428812-13:head [read 0~212992,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-04-26T01:08:42.524 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:29.783527+0000", 2024-04-26T01:08:42.524 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.690497179999999, 2024-04-26T01:08:42.524 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.37062857599999999, 2024-04-26T01:08:42.524 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.524 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.525 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:42.525 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:42.525 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:42.525 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:42.525 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 688 2024-04-26T01:08:42.525 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.525 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.525 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.525 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.525 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.783527+0000", 2024-04-26T01:08:42.525 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.526 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.526 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.526 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.526 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.783527+0000", 2024-04-26T01:08:42.526 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.526 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.526 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.526 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.526 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.783528+0000", 2024-04-26T01:08:42.526 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.187e-06 2024-04-26T01:08:42.526 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.526 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.526 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.527 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.783535+0000", 2024-04-26T01:08:42.527 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3130000000000004e-06 2024-04-26T01:08:42.527 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.527 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.527 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.527 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.783546+0000", 2024-04-26T01:08:42.527 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1093e-05 2024-04-26T01:08:42.527 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.527 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.527 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.527 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.783553+0000", 2024-04-26T01:08:42.527 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6899999999999992e-06 2024-04-26T01:08:42.527 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.527 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.528 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.528 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.785428+0000", 2024-04-26T01:08:42.528 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001874697 2024-04-26T01:08:42.528 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.528 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.528 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:42.528 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.785679+0000", 2024-04-26T01:08:42.528 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00025062800000000003 2024-04-26T01:08:42.528 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.528 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.528 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.528 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:30.154156+0000", 2024-04-26T01:08:42.528 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.36847696800000002 2024-04-26T01:08:42.528 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.528 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.529 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.529 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.529 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.529 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:762 3.fs0 3:fcac9f29:::smithi16428812-36:head [setxattr _header (57) in=7b] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:08:42.529 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.224739+0000", 2024-04-26T01:08:42.529 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 5.2492857449999999, 2024-04-26T01:08:42.529 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35577256200000001, 2024-04-26T01:08:42.529 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.529 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.529 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:42.529 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:42.529 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:42.529 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:42.529 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 762 2024-04-26T01:08:42.529 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.530 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.530 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.530 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.530 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224739+0000", 2024-04-26T01:08:42.530 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.530 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.530 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.530 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.530 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224739+0000", 2024-04-26T01:08:42.530 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.530 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.530 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.530 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.530 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224739+0000", 2024-04-26T01:08:42.531 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3e-07 2024-04-26T01:08:42.531 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.531 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.531 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.531 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224744+0000", 2024-04-26T01:08:42.531 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5920000000000002e-06 2024-04-26T01:08:42.531 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.531 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.531 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.531 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224751+0000", 2024-04-26T01:08:42.531 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6039999999999998e-06 2024-04-26T01:08:42.531 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.532 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.532 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.532 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224756+0000", 2024-04-26T01:08:42.532 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.006e-06 2024-04-26T01:08:42.532 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.532 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.532 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.532 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.228205+0000", 2024-04-26T01:08:42.532 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003449971 2024-04-26T01:08:42.532 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.532 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.532 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:42.532 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.228282+0000", 2024-04-26T01:08:42.533 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6625000000000006e-05 2024-04-26T01:08:42.533 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.533 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.533 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:42.533 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.228828+0000", 2024-04-26T01:08:42.533 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00054617499999999998 2024-04-26T01:08:42.533 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.533 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.533 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:42.533 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.230451+0000", 2024-04-26T01:08:42.533 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001622477 2024-04-26T01:08:42.533 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.533 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.533 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:42.533 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.580252+0000", 2024-04-26T01:08:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.34980141599999998 2024-04-26T01:08:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.580511+0000", 2024-04-26T01:08:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00025896600000000001 2024-04-26T01:08:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:763 3.fs0 3:fcac9f29:::smithi16428812-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:08:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.224763+0000", 2024-04-26T01:08:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 5.2492611949999999, 2024-04-26T01:08:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35969443000000001, 2024-04-26T01:08:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 763 2024-04-26T01:08:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224763+0000", 2024-04-26T01:08:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.536 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.536 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.536 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224763+0000", 2024-04-26T01:08:42.536 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.536 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.536 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.536 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.536 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224764+0000", 2024-04-26T01:08:42.536 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8999999999999996e-07 2024-04-26T01:08:42.536 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.536 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.536 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.536 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224767+0000", 2024-04-26T01:08:42.536 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.118e-06 2024-04-26T01:08:42.537 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.537 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.537 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.537 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224774+0000", 2024-04-26T01:08:42.537 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0840000000000003e-06 2024-04-26T01:08:42.537 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.537 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.537 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.537 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224779+0000", 2024-04-26T01:08:42.537 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.31e-06 2024-04-26T01:08:42.537 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.537 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.537 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.537 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.229405+0000", 2024-04-26T01:08:42.538 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046254920000000001 2024-04-26T01:08:42.538 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.538 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.538 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:42.538 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.229732+0000", 2024-04-26T01:08:42.538 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00032752299999999997 2024-04-26T01:08:42.538 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.538 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.538 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.538 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.581793+0000", 2024-04-26T01:08:42.538 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35206099899999999 2024-04-26T01:08:42.538 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.538 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.538 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:42.539 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.582075+0000", 2024-04-26T01:08:42.539 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00028167900000000001 2024-04-26T01:08:42.539 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.539 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.539 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.539 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.584458+0000", 2024-04-26T01:08:42.539 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0023826350000000001 2024-04-26T01:08:42.539 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.539 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.539 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.539 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.539 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.539 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 71/44 ECSubWriteReply(tid=420, last_complete=71'100, committed=1, applied=1))", 2024-04-26T01:08:42.539 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.229879+0000", 2024-04-26T01:08:42.540 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 5.2441453170000001, 2024-04-26T01:08:42.540 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35171478499999997, 2024-04-26T01:08:42.540 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.540 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.540 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:42.540 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.540 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.540 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.540 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.229879+0000", 2024-04-26T01:08:42.540 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.540 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.540 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.540 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.540 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.229879+0000", 2024-04-26T01:08:42.540 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.541 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.541 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.541 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.541 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.229880+0000", 2024-04-26T01:08:42.541 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1960000000000001e-06 2024-04-26T01:08:42.541 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.541 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.541 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.541 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.229888+0000", 2024-04-26T01:08:42.541 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5399999999999998e-06 2024-04-26T01:08:42.541 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.541 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.541 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.541 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.579761+0000", 2024-04-26T01:08:42.542 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.34987286400000001 2024-04-26T01:08:42.542 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.542 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.542 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.542 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.579780+0000", 2024-04-26T01:08:42.542 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9147000000000002e-05 2024-04-26T01:08:42.542 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.542 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.542 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.542 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.579993+0000", 2024-04-26T01:08:42.542 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000213599 2024-04-26T01:08:42.542 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.542 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.542 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.543 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.581594+0000", 2024-04-26T01:08:42.543 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0016004389999999999 2024-04-26T01:08:42.543 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.543 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.543 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.543 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.543 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.543 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:768 3.bs0 3:d8d6d69b:::smithi16428812-48:head [setxattr _header (57) in=7b] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:08:42.543 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.237741+0000", 2024-04-26T01:08:42.543 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 5.2362834520000003, 2024-04-26T01:08:42.543 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.36214788399999998, 2024-04-26T01:08:42.543 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.543 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.543 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:42.544 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:42.544 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:42.544 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:42.544 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 768 2024-04-26T01:08:42.544 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.544 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.544 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.544 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.544 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237741+0000", 2024-04-26T01:08:42.544 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.544 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.544 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.544 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.544 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237741+0000", 2024-04-26T01:08:42.545 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.545 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.545 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.545 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.545 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237742+0000", 2024-04-26T01:08:42.545 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4200000000000005e-07 2024-04-26T01:08:42.545 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.545 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.545 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.545 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237747+0000", 2024-04-26T01:08:42.545 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.921e-06 2024-04-26T01:08:42.545 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.545 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.545 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.546 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237757+0000", 2024-04-26T01:08:42.546 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0068000000000001e-05 2024-04-26T01:08:42.546 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.546 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.546 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.546 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237764+0000", 2024-04-26T01:08:42.546 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9689999999999997e-06 2024-04-26T01:08:42.546 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.546 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.546 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.546 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.241404+0000", 2024-04-26T01:08:42.546 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0036398590000000001 2024-04-26T01:08:42.546 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.547 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.547 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:42.547 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.241528+0000", 2024-04-26T01:08:42.547 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000124258 2024-04-26T01:08:42.547 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.547 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.547 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:42.547 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.242111+0000", 2024-04-26T01:08:42.547 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00058331400000000003 2024-04-26T01:08:42.547 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.547 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.547 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:42.547 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.242834+0000", 2024-04-26T01:08:42.547 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00072257799999999998 2024-04-26T01:08:42.547 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.548 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.548 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:42.548 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.599608+0000", 2024-04-26T01:08:42.548 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35677461399999999 2024-04-26T01:08:42.548 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.548 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.548 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.548 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.599889+0000", 2024-04-26T01:08:42.548 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00028046100000000002 2024-04-26T01:08:42.548 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.548 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.548 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.548 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.548 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.548 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:769 3.bs0 3:d8d6d69b:::smithi16428812-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:08:42.549 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.237777+0000", 2024-04-26T01:08:42.549 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 5.2362473769999998, 2024-04-26T01:08:42.549 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.840927439, 2024-04-26T01:08:42.549 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.549 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.549 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:42.549 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:42.549 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:42.549 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:42.549 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 769 2024-04-26T01:08:42.549 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.549 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.550 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.550 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.550 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237777+0000", 2024-04-26T01:08:42.550 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.550 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.550 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.550 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.550 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237777+0000", 2024-04-26T01:08:42.550 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.550 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.550 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.550 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.550 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237778+0000", 2024-04-26T01:08:42.550 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.92e-07 2024-04-26T01:08:42.551 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.551 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.551 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.551 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237782+0000", 2024-04-26T01:08:42.551 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7029999999999997e-06 2024-04-26T01:08:42.551 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.551 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.551 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.551 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237789+0000", 2024-04-26T01:08:42.551 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.063e-06 2024-04-26T01:08:42.551 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.551 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.551 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.551 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237796+0000", 2024-04-26T01:08:42.551 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1330000000000004e-06 2024-04-26T01:08:42.552 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.552 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.552 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.552 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.242751+0000", 2024-04-26T01:08:42.552 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0049552069999999997 2024-04-26T01:08:42.552 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.552 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.552 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:42.552 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.242818+0000", 2024-04-26T01:08:42.553 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7502999999999994e-05 2024-04-26T01:08:42.553 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.553 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.553 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.553 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.601011+0000", 2024-04-26T01:08:42.553 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.358193066 2024-04-26T01:08:42.553 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.553 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.553 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:42.553 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.601277+0000", 2024-04-26T01:08:42.554 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026577300000000002 2024-04-26T01:08:42.554 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.554 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.554 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.554 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.078704+0000", 2024-04-26T01:08:42.554 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.47742729900000003 2024-04-26T01:08:42.554 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.554 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.554 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.554 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.554 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.554 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 71/30 ECSubWriteReply(tid=425, last_complete=71'63, committed=1, applied=1))", 2024-04-26T01:08:42.554 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.243053+0000", 2024-04-26T01:08:42.554 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 5.230971394, 2024-04-26T01:08:42.555 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35781853899999999, 2024-04-26T01:08:42.555 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.555 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.555 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:42.555 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.555 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.555 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.555 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.243053+0000", 2024-04-26T01:08:42.555 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.555 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.555 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.555 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.555 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.243053+0000", 2024-04-26T01:08:42.555 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.556 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.556 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.556 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.556 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.243053+0000", 2024-04-26T01:08:42.556 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.0800000000000005e-07 2024-04-26T01:08:42.556 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.556 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.556 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.556 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.243057+0000", 2024-04-26T01:08:42.556 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4309999999999998e-06 2024-04-26T01:08:42.556 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.556 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.556 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.556 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.598402+0000", 2024-04-26T01:08:42.557 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35534554099999999 2024-04-26T01:08:42.557 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.557 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.557 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.557 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.598415+0000", 2024-04-26T01:08:42.557 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.279e-05 2024-04-26T01:08:42.557 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.557 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.557 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.557 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.599361+0000", 2024-04-26T01:08:42.557 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00094610600000000005 2024-04-26T01:08:42.557 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.557 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.558 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.558 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.600871+0000", 2024-04-26T01:08:42.558 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015101629999999999 2024-04-26T01:08:42.558 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.558 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.558 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.558 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.558 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.558 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 71/44 ECSubWriteReply(tid=426, last_complete=71'100, committed=1, applied=1))", 2024-04-26T01:08:42.558 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.582864+0000", 2024-04-26T01:08:42.558 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.8911607540000004, 2024-04-26T01:08:42.558 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.42299990500000001, 2024-04-26T01:08:42.558 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.559 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.559 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:42.559 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.559 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.559 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.559 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.582864+0000", 2024-04-26T01:08:42.559 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.559 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.559 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.559 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.559 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.582864+0000", 2024-04-26T01:08:42.559 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.559 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.559 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.559 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.582865+0000", 2024-04-26T01:08:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0899999999999999e-06 2024-04-26T01:08:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.582874+0000", 2024-04-26T01:08:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9360000000000005e-06 2024-04-26T01:08:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.005609+0000", 2024-04-26T01:08:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.422735317 2024-04-26T01:08:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.561 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.005623+0000", 2024-04-26T01:08:42.561 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.394e-05 2024-04-26T01:08:42.561 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.561 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.561 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.561 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.005793+0000", 2024-04-26T01:08:42.561 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000170142 2024-04-26T01:08:42.561 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.561 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.561 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.561 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.005863+0000", 2024-04-26T01:08:42.561 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0480000000000003e-05 2024-04-26T01:08:42.561 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.561 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.562 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.562 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.562 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.562 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 71/30 ECSubWriteReply(tid=428, last_complete=71'63, committed=1, applied=1))", 2024-04-26T01:08:42.562 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.601257+0000", 2024-04-26T01:08:42.562 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.8727674069999996, 2024-04-26T01:08:42.562 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.99078997400000002, 2024-04-26T01:08:42.562 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.562 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.562 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:42.562 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.562 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.562 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.562 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.601257+0000", 2024-04-26T01:08:42.563 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.563 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.563 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.563 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.563 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.601257+0000", 2024-04-26T01:08:42.563 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.563 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.563 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.563 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.563 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.601259+0000", 2024-04-26T01:08:42.563 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.59e-06 2024-04-26T01:08:42.563 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.563 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.563 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.564 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.601268+0000", 2024-04-26T01:08:42.564 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.38e-06 2024-04-26T01:08:42.564 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.564 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.564 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.564 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.591752+0000", 2024-04-26T01:08:42.564 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.990484162 2024-04-26T01:08:42.564 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.564 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.564 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.564 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.591768+0000", 2024-04-26T01:08:42.564 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6169999999999999e-05 2024-04-26T01:08:42.564 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.564 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.565 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.565 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.591964+0000", 2024-04-26T01:08:42.565 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019577900000000001 2024-04-26T01:08:42.565 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.565 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.565 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.565 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.592047+0000", 2024-04-26T01:08:42.565 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2892999999999994e-05 2024-04-26T01:08:42.565 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.565 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.565 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.565 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.565 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.565 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpReadReply(3.bs0 71/30 ECSubReadReply(tid=429, attrs_read=0))", 2024-04-26T01:08:42.566 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.602946+0000", 2024-04-26T01:08:42.566 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.871078539, 2024-04-26T01:08:42.566 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.47577750400000002, 2024-04-26T01:08:42.566 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:42.566 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:42.566 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:42.566 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:42.566 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.566 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:42.566 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.602946+0000", 2024-04-26T01:08:42.566 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.566 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.566 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.567 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:42.567 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.602946+0000", 2024-04-26T01:08:42.567 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:42.567 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.567 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.567 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:42.567 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.602947+0000", 2024-04-26T01:08:42.567 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2699999999999998e-07 2024-04-26T01:08:42.567 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.567 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.567 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:42.567 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.602961+0000", 2024-04-26T01:08:42.567 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.485e-05 2024-04-26T01:08:42.567 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.568 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.568 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:42.568 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.078257+0000", 2024-04-26T01:08:42.568 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.47529542299999999 2024-04-26T01:08:42.568 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.568 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.568 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:42.568 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.078275+0000", 2024-04-26T01:08:42.568 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7796e-05 2024-04-26T01:08:42.568 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.568 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.568 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:42.568 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.078412+0000", 2024-04-26T01:08:42.569 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013743300000000001 2024-04-26T01:08:42.569 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:42.569 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:42.569 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:42.569 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.078723+0000", 2024-04-26T01:08:42.569 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00031117500000000001 2024-04-26T01:08:42.569 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.569 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.569 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.569 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:42.569 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:42.569 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:42.571 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:08:42.571 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:42.565+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:42.672 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:42.669+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:42.705 INFO:tasks.rados.rados.0.smithi164.stdout:330: done (2 left) 2024-04-26T01:08:42.705 INFO:tasks.rados.rados.0.smithi164.stdout:331: done (1 left) 2024-04-26T01:08:42.705 INFO:tasks.rados.rados.0.smithi164.stdout:332: done (0 left) 2024-04-26T01:08:42.705 INFO:tasks.rados.rados.0.smithi164.stdout:333: copy_from oid 25 from oid 11 current snap is 22 2024-04-26T01:08:42.706 INFO:tasks.rados.rados.0.smithi164.stdout:334: snap_create 2024-04-26T01:08:42.726 INFO:tasks.rados.rados.0.smithi164.stdout:333: finishing copy_from racing read to smithi16428812-25 2024-04-26T01:08:42.728 INFO:tasks.rados.rados.0.smithi164.stdout:333: finishing copy_from to smithi16428812-25 2024-04-26T01:08:42.728 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 25 v 37 (ObjNum 138 snap 21 seq_num 138) dirty exists 2024-04-26T01:08:42.747 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:42.747 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:42.748 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-04-26T01:08:42.748 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:42.760 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:08:42.773 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:42.771+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:42.874 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:42.872+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:42.885 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:42.885 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:42.885 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-04-26T01:08:42.885 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-04-26T01:08:42.885 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:42.901 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:08:42.975 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:42.975+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:43.072 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:43.072 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:43.072 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:43.073 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:43.073 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.073 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:16 3.4s0 3:2026e263:::smithi16428812-6:head [append 0~98304,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:43.073 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.307585+0000", 2024-04-26T01:08:43.073 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 43.761625404999997, 2024-04-26T01:08:43.073 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.016360486, 2024-04-26T01:08:43.073 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.073 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.073 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:43.073 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.073 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.073 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.073 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 16 2024-04-26T01:08:43.073 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.074 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.074 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.074 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.074 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307585+0000", 2024-04-26T01:08:43.074 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.075 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.075 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307585+0000", 2024-04-26T01:08:43.075 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.075 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.075 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307587+0000", 2024-04-26T01:08:43.075 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1280000000000002e-06 2024-04-26T01:08:43.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.075 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.076 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308008+0000", 2024-04-26T01:08:43.076 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00042080399999999999 2024-04-26T01:08:43.076 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.076 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.076 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.076 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308031+0000", 2024-04-26T01:08:43.076 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3085999999999999e-05 2024-04-26T01:08:43.076 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.076 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.076 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.076 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308041+0000", 2024-04-26T01:08:43.076 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9650000000000003e-06 2024-04-26T01:08:43.076 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308110+0000", 2024-04-26T01:08:43.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.915e-05 2024-04-26T01:08:43.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:43.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308277+0000", 2024-04-26T01:08:43.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016695800000000001 2024-04-26T01:08:43.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.309191+0000", 2024-04-26T01:08:43.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00091407599999999999 2024-04-26T01:08:43.078 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.078 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.078 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.078 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314046+0000", 2024-04-26T01:08:43.078 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.004854377 2024-04-26T01:08:43.078 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.078 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.078 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:43.078 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323807+0000", 2024-04-26T01:08:43.078 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0097610869999999999 2024-04-26T01:08:43.078 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.078 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.078 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.078 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323946+0000", 2024-04-26T01:08:43.079 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013885500000000001 2024-04-26T01:08:43.079 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.079 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.079 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.079 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.079 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.079 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:17 3.4s0 3:2026e263:::smithi16428812-6:head [setxattr _header (55) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:43.079 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.335986+0000", 2024-04-26T01:08:43.079 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 43.733224198999999, 2024-04-26T01:08:43.079 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010264923, 2024-04-26T01:08:43.079 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.079 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.080 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:43.080 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.080 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.080 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.080 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 17 2024-04-26T01:08:43.080 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.080 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.080 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.080 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.080 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.335986+0000", 2024-04-26T01:08:43.080 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.080 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.080 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.080 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.081 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.335986+0000", 2024-04-26T01:08:43.081 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.081 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.081 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.081 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.081 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.335988+0000", 2024-04-26T01:08:43.081 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.373e-06 2024-04-26T01:08:43.081 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.081 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.081 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.082 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.335994+0000", 2024-04-26T01:08:43.082 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0700000000000003e-06 2024-04-26T01:08:43.082 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.082 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.082 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.082 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336007+0000", 2024-04-26T01:08:43.082 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3577000000000001e-05 2024-04-26T01:08:43.082 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.082 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.082 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.082 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336018+0000", 2024-04-26T01:08:43.082 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0668e-05 2024-04-26T01:08:43.082 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.083 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.083 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.083 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336285+0000", 2024-04-26T01:08:43.083 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026682799999999999 2024-04-26T01:08:43.083 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.083 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.083 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:43.083 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336439+0000", 2024-04-26T01:08:43.083 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000154048 2024-04-26T01:08:43.083 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.083 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.083 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.083 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.338115+0000", 2024-04-26T01:08:43.083 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0016757880000000001 2024-04-26T01:08:43.083 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.084 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.084 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.084 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.339187+0000", 2024-04-26T01:08:43.084 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001072494 2024-04-26T01:08:43.084 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.084 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.084 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:43.084 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.346146+0000", 2024-04-26T01:08:43.084 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0069590279999999999 2024-04-26T01:08:43.084 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.084 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.084 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.084 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.346251+0000", 2024-04-26T01:08:43.084 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000105049 2024-04-26T01:08:43.085 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.085 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.085 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.085 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.085 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.085 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:18 3.4s0 3:2026e263:::smithi16428812-6:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:43.085 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.336046+0000", 2024-04-26T01:08:43.085 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 43.733164275, 2024-04-26T01:08:43.085 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013179965, 2024-04-26T01:08:43.085 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.085 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.086 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:43.086 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.086 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.086 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.086 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 18 2024-04-26T01:08:43.086 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.086 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.086 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.086 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.086 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336046+0000", 2024-04-26T01:08:43.086 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.086 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.086 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.086 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.087 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336046+0000", 2024-04-26T01:08:43.087 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.087 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.087 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.087 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.087 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336047+0000", 2024-04-26T01:08:43.087 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1119999999999999e-06 2024-04-26T01:08:43.087 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.087 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.087 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.087 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336052+0000", 2024-04-26T01:08:43.087 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3510000000000004e-06 2024-04-26T01:08:43.087 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.087 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.088 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.088 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336063+0000", 2024-04-26T01:08:43.088 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1722999999999999e-05 2024-04-26T01:08:43.088 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.088 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.088 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.088 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336070+0000", 2024-04-26T01:08:43.088 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6259999999999997e-06 2024-04-26T01:08:43.088 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.088 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.088 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.088 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.338893+0000", 2024-04-26T01:08:43.088 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0028230009999999999 2024-04-26T01:08:43.088 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.089 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.089 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:43.089 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.339060+0000", 2024-04-26T01:08:43.089 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016734700000000001 2024-04-26T01:08:43.089 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.089 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.089 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.089 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.346619+0000", 2024-04-26T01:08:43.089 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0075586949999999998 2024-04-26T01:08:43.089 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.089 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.090 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:43.090 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.346731+0000", 2024-04-26T01:08:43.090 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000111863 2024-04-26T01:08:43.090 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.090 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.090 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.090 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349226+0000", 2024-04-26T01:08:43.090 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024952469999999999 2024-04-26T01:08:43.090 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.090 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.090 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.090 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.090 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.091 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.5s1 42/30 ECSubWrite(tid=32, reqid=client.4521.0:351, at_version=42'28, trim_to=0'0, roll_forward_to=42'25))", 2024-04-26T01:08:43.091 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:08.417770+0000", 2024-04-26T01:08:43.091 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 34.651440407000003, 2024-04-26T01:08:43.091 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.64597556300000003, 2024-04-26T01:08:43.091 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.091 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.091 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.091 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.091 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.091 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.091 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.417770+0000", 2024-04-26T01:08:43.091 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.092 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.092 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.092 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.092 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.417770+0000", 2024-04-26T01:08:43.092 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.092 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.092 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.092 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.092 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.417771+0000", 2024-04-26T01:08:43.092 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6499999999999998e-07 2024-04-26T01:08:43.093 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.093 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.093 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.093 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.417789+0000", 2024-04-26T01:08:43.093 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8198000000000001e-05 2024-04-26T01:08:43.093 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.093 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.093 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.093 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.062086+0000", 2024-04-26T01:08:43.093 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.64429725800000004 2024-04-26T01:08:43.093 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.093 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.094 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.094 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.062104+0000", 2024-04-26T01:08:43.094 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8053000000000001e-05 2024-04-26T01:08:43.094 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.094 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.094 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.094 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.062287+0000", 2024-04-26T01:08:43.094 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00018218499999999999 2024-04-26T01:08:43.094 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.094 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.094 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.094 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.062316+0000", 2024-04-26T01:08:43.094 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9547e-05 2024-04-26T01:08:43.094 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.095 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.095 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.095 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.063675+0000", 2024-04-26T01:08:43.095 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001358831 2024-04-26T01:08:43.096 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.096 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.096 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.096 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.063746+0000", 2024-04-26T01:08:43.096 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0625999999999998e-05 2024-04-26T01:08:43.096 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.096 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.096 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.096 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.096 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.096 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(osd.14.21:2 3.cs1 3.83402d7c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:43.096 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.432903+0000", 2024-04-26T01:08:43.096 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 33.636307846000001, 2024-04-26T01:08:43.096 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97310677899999998, 2024-04-26T01:08:43.097 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.097 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.097 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.097 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.097 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.097 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.097 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432903+0000", 2024-04-26T01:08:43.097 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.097 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.097 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.097 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.097 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432903+0000", 2024-04-26T01:08:43.097 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.097 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.097 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.098 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.098 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432904+0000", 2024-04-26T01:08:43.098 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4839999999999999e-06 2024-04-26T01:08:43.098 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.098 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.098 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.098 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432912+0000", 2024-04-26T01:08:43.098 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3489999999999992e-06 2024-04-26T01:08:43.098 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.098 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.099 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.099 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432927+0000", 2024-04-26T01:08:43.099 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4297000000000001e-05 2024-04-26T01:08:43.099 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.099 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.099 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.099 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432935+0000", 2024-04-26T01:08:43.099 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0360000000000006e-06 2024-04-26T01:08:43.099 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.100 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.100 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.100 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432997+0000", 2024-04-26T01:08:43.100 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2176999999999998e-05 2024-04-26T01:08:43.100 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.100 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.100 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:43.100 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.433000+0000", 2024-04-26T01:08:43.100 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1990000000000002e-06 2024-04-26T01:08:43.101 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.101 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.101 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.101 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.405993+0000", 2024-04-26T01:08:43.101 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97299321100000002 2024-04-26T01:08:43.101 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.101 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.101 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.101 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406009+0000", 2024-04-26T01:08:43.101 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6025999999999999e-05 2024-04-26T01:08:43.101 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.101 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.101 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.101 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.101 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.102 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:375 3.cs1 3.33e19ddc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:43.102 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.436798+0000", 2024-04-26T01:08:43.102 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 33.632412451, 2024-04-26T01:08:43.102 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96902882700000004, 2024-04-26T01:08:43.102 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.102 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.102 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.102 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.102 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.102 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.102 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 375 2024-04-26T01:08:43.102 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.103 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.103 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.103 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.103 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436798+0000", 2024-04-26T01:08:43.103 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.103 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.103 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.103 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.103 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436798+0000", 2024-04-26T01:08:43.103 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.103 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.103 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.104 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.104 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436800+0000", 2024-04-26T01:08:43.104 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.756e-06 2024-04-26T01:08:43.104 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.104 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.104 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.104 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436810+0000", 2024-04-26T01:08:43.104 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9860000000000006e-06 2024-04-26T01:08:43.104 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.104 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.104 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.104 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436834+0000", 2024-04-26T01:08:43.104 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4457e-05 2024-04-26T01:08:43.104 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.105 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.105 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.105 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436846+0000", 2024-04-26T01:08:43.105 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1963999999999999e-05 2024-04-26T01:08:43.105 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.105 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.105 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.105 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436916+0000", 2024-04-26T01:08:43.105 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9667999999999998e-05 2024-04-26T01:08:43.105 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.105 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.105 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:43.105 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436918+0000", 2024-04-26T01:08:43.106 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.294e-06 2024-04-26T01:08:43.106 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.106 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.106 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.106 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.405810+0000", 2024-04-26T01:08:43.106 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96889232599999997 2024-04-26T01:08:43.106 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.106 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.106 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.106 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.405827+0000", 2024-04-26T01:08:43.106 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6376e-05 2024-04-26T01:08:43.106 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.106 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.106 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.107 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.107 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.107 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:376 3.cs1 3.33e19ddc (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:43.107 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.436879+0000", 2024-04-26T01:08:43.107 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 33.632331114000003, 2024-04-26T01:08:43.107 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96918375199999995, 2024-04-26T01:08:43.107 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.107 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.107 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.107 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.107 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.107 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.107 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 376 2024-04-26T01:08:43.107 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.108 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.108 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.108 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.108 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436879+0000", 2024-04-26T01:08:43.108 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.108 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.108 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.108 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.108 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436879+0000", 2024-04-26T01:08:43.108 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.109 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.109 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.109 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.109 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436880+0000", 2024-04-26T01:08:43.109 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0100000000000004e-07 2024-04-26T01:08:43.109 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.109 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.109 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.109 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436885+0000", 2024-04-26T01:08:43.109 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3959999999999999e-06 2024-04-26T01:08:43.109 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.110 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.110 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.110 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436896+0000", 2024-04-26T01:08:43.110 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1291000000000001e-05 2024-04-26T01:08:43.110 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.110 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.110 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.110 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436904+0000", 2024-04-26T01:08:43.110 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6000000000000007e-06 2024-04-26T01:08:43.110 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.110 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.110 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.110 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436935+0000", 2024-04-26T01:08:43.110 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0626000000000001e-05 2024-04-26T01:08:43.111 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.111 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.111 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:43.111 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436936+0000", 2024-04-26T01:08:43.111 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.44e-07 2024-04-26T01:08:43.111 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.111 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.111 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.111 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406050+0000", 2024-04-26T01:08:43.111 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96911480699999997 2024-04-26T01:08:43.111 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.111 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.111 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.111 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406063+0000", 2024-04-26T01:08:43.112 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2687000000000001e-05 2024-04-26T01:08:43.112 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.112 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.112 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.112 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.112 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.112 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:380 3.cs1 3.f9ba2d2c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:43.112 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.444825+0000", 2024-04-26T01:08:43.112 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 33.624385451999999, 2024-04-26T01:08:43.112 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96128573799999995, 2024-04-26T01:08:43.112 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.112 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.112 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.112 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.113 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.113 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.113 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 380 2024-04-26T01:08:43.113 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.113 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.113 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.113 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.113 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.444825+0000", 2024-04-26T01:08:43.113 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.113 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.113 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.113 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.113 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.444825+0000", 2024-04-26T01:08:43.113 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.114 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.114 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.114 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.114 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.444827+0000", 2024-04-26T01:08:43.114 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.731e-06 2024-04-26T01:08:43.114 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.114 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.114 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.114 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.445681+0000", 2024-04-26T01:08:43.114 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00085380000000000005 2024-04-26T01:08:43.114 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.114 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.114 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.114 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.445706+0000", 2024-04-26T01:08:43.115 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5768000000000001e-05 2024-04-26T01:08:43.115 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.115 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.115 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.115 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.445720+0000", 2024-04-26T01:08:43.115 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3788000000000001e-05 2024-04-26T01:08:43.115 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.115 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.115 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.115 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.445801+0000", 2024-04-26T01:08:43.115 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1048000000000001e-05 2024-04-26T01:08:43.115 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.115 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.115 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:43.116 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.445805+0000", 2024-04-26T01:08:43.116 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7479999999999999e-06 2024-04-26T01:08:43.116 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.116 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.116 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.116 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406098+0000", 2024-04-26T01:08:43.116 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96029355199999999 2024-04-26T01:08:43.116 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.116 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.116 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.116 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406111+0000", 2024-04-26T01:08:43.116 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2303000000000001e-05 2024-04-26T01:08:43.116 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.116 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.117 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.117 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.117 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.117 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:381 3.cs1 3.f9ba2d2c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:43.117 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.450315+0000", 2024-04-26T01:08:43.117 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 33.618895135999999, 2024-04-26T01:08:43.117 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95584751300000004, 2024-04-26T01:08:43.117 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.117 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.117 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.117 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.117 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.117 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.117 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 381 2024-04-26T01:08:43.117 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.118 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.118 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.118 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.118 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.450315+0000", 2024-04-26T01:08:43.118 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.118 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.118 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.118 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.118 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.450315+0000", 2024-04-26T01:08:43.118 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.118 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.118 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.118 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.118 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.450317+0000", 2024-04-26T01:08:43.119 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.271e-06 2024-04-26T01:08:43.119 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.119 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.119 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.119 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451037+0000", 2024-04-26T01:08:43.119 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00071991199999999996 2024-04-26T01:08:43.119 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.119 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.119 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.119 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451053+0000", 2024-04-26T01:08:43.119 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6049000000000002e-05 2024-04-26T01:08:43.119 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.119 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.120 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.120 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451060+0000", 2024-04-26T01:08:43.120 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8839999999999994e-06 2024-04-26T01:08:43.120 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.120 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.120 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.120 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451144+0000", 2024-04-26T01:08:43.120 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3251999999999995e-05 2024-04-26T01:08:43.120 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.120 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.120 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:43.120 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451147+0000", 2024-04-26T01:08:43.120 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.994e-06 2024-04-26T01:08:43.120 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.121 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.121 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.121 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406150+0000", 2024-04-26T01:08:43.121 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95500295999999996 2024-04-26T01:08:43.121 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.121 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.121 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.121 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406163+0000", 2024-04-26T01:08:43.121 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3191e-05 2024-04-26T01:08:43.121 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.121 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.121 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.121 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.121 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.122 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:382 3.cs1 3.f9ba2d2c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:43.122 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.451079+0000", 2024-04-26T01:08:43.122 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 33.618131398999999, 2024-04-26T01:08:43.122 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95513123099999997, 2024-04-26T01:08:43.122 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.122 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.122 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.122 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.122 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.122 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.122 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 382 2024-04-26T01:08:43.122 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.122 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.122 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.123 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.123 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451079+0000", 2024-04-26T01:08:43.123 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.123 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.123 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.123 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.123 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451079+0000", 2024-04-26T01:08:43.123 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.123 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.123 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.123 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.123 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451080+0000", 2024-04-26T01:08:43.123 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9599999999999999e-07 2024-04-26T01:08:43.123 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.124 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.124 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.124 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451116+0000", 2024-04-26T01:08:43.124 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6797e-05 2024-04-26T01:08:43.124 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.124 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.124 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.124 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451137+0000", 2024-04-26T01:08:43.124 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0163e-05 2024-04-26T01:08:43.124 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.124 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.124 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.124 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451142+0000", 2024-04-26T01:08:43.124 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6450000000000002e-06 2024-04-26T01:08:43.125 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.125 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.125 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.125 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451200+0000", 2024-04-26T01:08:43.125 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7703e-05 2024-04-26T01:08:43.125 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.125 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.125 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:43.125 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451201+0000", 2024-04-26T01:08:43.125 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1200000000000001e-07 2024-04-26T01:08:43.125 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.125 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.125 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.125 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406198+0000", 2024-04-26T01:08:43.126 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95499736000000002 2024-04-26T01:08:43.126 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.126 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.126 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.126 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406210+0000", 2024-04-26T01:08:43.126 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2055000000000001e-05 2024-04-26T01:08:43.126 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.126 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.126 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.126 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.126 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.126 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:383 3.cs1 3.f9ba2d2c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:43.126 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.451164+0000", 2024-04-26T01:08:43.127 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 33.618046036999999, 2024-04-26T01:08:43.127 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95510495399999995, 2024-04-26T01:08:43.127 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.127 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.127 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.127 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.127 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.127 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.127 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 383 2024-04-26T01:08:43.127 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.127 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.127 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.127 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.127 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451164+0000", 2024-04-26T01:08:43.127 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.128 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.128 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.128 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.128 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451164+0000", 2024-04-26T01:08:43.128 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.128 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.128 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.128 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.128 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451165+0000", 2024-04-26T01:08:43.128 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.58e-07 2024-04-26T01:08:43.128 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.128 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.128 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.128 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451168+0000", 2024-04-26T01:08:43.128 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3000000000000002e-06 2024-04-26T01:08:43.129 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.129 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.129 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.129 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451174+0000", 2024-04-26T01:08:43.129 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8220000000000003e-06 2024-04-26T01:08:43.129 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.129 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.129 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.129 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451179+0000", 2024-04-26T01:08:43.129 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4700000000000004e-06 2024-04-26T01:08:43.129 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.129 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.129 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.129 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451233+0000", 2024-04-26T01:08:43.129 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4086999999999999e-05 2024-04-26T01:08:43.130 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.130 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.130 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:43.130 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451234+0000", 2024-04-26T01:08:43.130 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0580000000000001e-06 2024-04-26T01:08:43.130 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.130 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.130 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.130 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406253+0000", 2024-04-26T01:08:43.130 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95501902800000005 2024-04-26T01:08:43.130 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.130 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.130 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.130 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406269+0000", 2024-04-26T01:08:43.131 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6731e-05 2024-04-26T01:08:43.131 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.131 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.131 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.131 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.131 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.131 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:384 3.cs1 3.f9ba2d2c (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:43.131 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.451185+0000", 2024-04-26T01:08:43.131 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 33.618025363000001, 2024-04-26T01:08:43.131 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95528391300000004, 2024-04-26T01:08:43.131 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.131 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.131 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.131 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.132 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.132 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.132 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 384 2024-04-26T01:08:43.132 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.132 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.132 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.132 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.132 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451185+0000", 2024-04-26T01:08:43.132 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.132 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.132 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.132 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.132 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451185+0000", 2024-04-26T01:08:43.132 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451186+0000", 2024-04-26T01:08:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3599999999999999e-07 2024-04-26T01:08:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451187+0000", 2024-04-26T01:08:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.804e-06 2024-04-26T01:08:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451193+0000", 2024-04-26T01:08:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4060000000000004e-06 2024-04-26T01:08:43.134 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.134 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.134 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.134 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451197+0000", 2024-04-26T01:08:43.134 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.262e-06 2024-04-26T01:08:43.134 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.134 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.134 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.134 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451254+0000", 2024-04-26T01:08:43.134 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6635000000000002e-05 2024-04-26T01:08:43.134 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.134 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.135 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:43.135 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451254+0000", 2024-04-26T01:08:43.135 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6599999999999995e-07 2024-04-26T01:08:43.135 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.135 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.135 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.135 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406454+0000", 2024-04-26T01:08:43.135 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95519958699999996 2024-04-26T01:08:43.135 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.135 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.135 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.135 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406469+0000", 2024-04-26T01:08:43.135 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5017e-05 2024-04-26T01:08:43.135 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.135 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.136 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.136 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.136 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.136 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:483 3.4s0 3:2d34bde3:::smithi16428812-29:head [copy-from ver 0] snapc a=[a,8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-26T01:08:43.136 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:14.042718+0000", 2024-04-26T01:08:43.136 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 29.026492791999999, 2024-04-26T01:08:43.136 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015567493, 2024-04-26T01:08:43.136 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.136 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.136 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:43.136 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.136 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.136 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.136 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 483 2024-04-26T01:08:43.137 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.137 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.137 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.137 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.137 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042718+0000", 2024-04-26T01:08:43.137 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.137 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.137 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.137 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.137 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042718+0000", 2024-04-26T01:08:43.137 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.137 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.137 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.137 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.138 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042721+0000", 2024-04-26T01:08:43.138 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.2100000000000002e-06 2024-04-26T01:08:43.138 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.138 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.138 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.138 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042756+0000", 2024-04-26T01:08:43.138 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4972000000000003e-05 2024-04-26T01:08:43.138 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.138 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.138 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.138 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042793+0000", 2024-04-26T01:08:43.138 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7027000000000003e-05 2024-04-26T01:08:43.138 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.138 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.139 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.139 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042819+0000", 2024-04-26T01:08:43.139 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6359999999999998e-05 2024-04-26T01:08:43.139 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.139 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.139 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.139 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042991+0000", 2024-04-26T01:08:43.139 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017123400000000001 2024-04-26T01:08:43.139 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.139 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.139 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op must wait for map", 2024-04-26T01:08:43.139 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.043005+0000", 2024-04-26T01:08:43.139 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4414e-05 2024-04-26T01:08:43.139 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.140 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.140 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.140 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.047524+0000", 2024-04-26T01:08:43.140 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0045195349999999999 2024-04-26T01:08:43.140 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.140 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.140 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:43.140 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.047801+0000", 2024-04-26T01:08:43.140 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00027666799999999999 2024-04-26T01:08:43.140 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.140 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.140 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.140 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.058285+0000", 2024-04-26T01:08:43.140 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010484073 2024-04-26T01:08:43.140 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.141 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.141 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.141 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.141 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.141 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:484 3.4s0 3:2d34bde3:::smithi16428812-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-26T01:08:43.141 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:14.042874+0000", 2024-04-26T01:08:43.141 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 29.026336728, 2024-04-26T01:08:43.141 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015940147000000002, 2024-04-26T01:08:43.141 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.141 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.141 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:43.141 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.141 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.141 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.142 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 484 2024-04-26T01:08:43.142 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.142 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.142 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.142 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.142 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042874+0000", 2024-04-26T01:08:43.142 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.142 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.142 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.142 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.142 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042874+0000", 2024-04-26T01:08:43.142 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.142 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.142 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.142 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.143 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042875+0000", 2024-04-26T01:08:43.143 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.64e-06 2024-04-26T01:08:43.143 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.143 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.143 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.143 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042884+0000", 2024-04-26T01:08:43.143 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1100000000000003e-06 2024-04-26T01:08:43.143 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.143 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.143 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.143 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042909+0000", 2024-04-26T01:08:43.143 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5857e-05 2024-04-26T01:08:43.143 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.143 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.144 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.144 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042929+0000", 2024-04-26T01:08:43.144 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9167000000000001e-05 2024-04-26T01:08:43.144 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.144 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.144 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.144 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.043135+0000", 2024-04-26T01:08:43.144 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00020666799999999999 2024-04-26T01:08:43.144 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.144 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.144 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:43.144 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.043158+0000", 2024-04-26T01:08:43.144 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.304e-05 2024-04-26T01:08:43.144 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.144 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.145 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.145 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.048261+0000", 2024-04-26T01:08:43.145 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0051031039999999998 2024-04-26T01:08:43.145 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.145 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.145 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for blocked object", 2024-04-26T01:08:43.145 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.048407+0000", 2024-04-26T01:08:43.145 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000145308 2024-04-26T01:08:43.145 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.145 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.145 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.145 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.058472+0000", 2024-04-26T01:08:43.145 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010065703 2024-04-26T01:08:43.145 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.146 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.146 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:43.146 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.058641+0000", 2024-04-26T01:08:43.146 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016832199999999999 2024-04-26T01:08:43.146 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.146 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.146 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.146 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.058814+0000", 2024-04-26T01:08:43.146 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017322799999999999 2024-04-26T01:08:43.146 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.146 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.146 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.146 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.146 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.146 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:605 3.4s0 3:2d34bde3:::smithi16428812-29:head [read 0~2129920,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:43.147 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.340722+0000", 2024-04-26T01:08:43.147 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 18.728488048999999, 2024-04-26T01:08:43.147 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010360348, 2024-04-26T01:08:43.147 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.147 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.147 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:43.147 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.147 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.147 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.147 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 605 2024-04-26T01:08:43.147 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.147 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.147 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.148 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.148 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340722+0000", 2024-04-26T01:08:43.148 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.148 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.148 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.148 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.148 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340722+0000", 2024-04-26T01:08:43.148 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.148 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.148 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.148 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.148 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340726+0000", 2024-04-26T01:08:43.148 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.467e-06 2024-04-26T01:08:43.148 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.148 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.149 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.149 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340743+0000", 2024-04-26T01:08:43.149 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7567e-05 2024-04-26T01:08:43.149 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.149 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.149 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.149 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340789+0000", 2024-04-26T01:08:43.149 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5259999999999997e-05 2024-04-26T01:08:43.149 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.149 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.149 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.149 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340812+0000", 2024-04-26T01:08:43.149 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2861999999999999e-05 2024-04-26T01:08:43.149 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.150 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.150 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.150 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340971+0000", 2024-04-26T01:08:43.150 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000159252 2024-04-26T01:08:43.150 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.150 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.150 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:43.150 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341112+0000", 2024-04-26T01:08:43.150 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000141349 2024-04-26T01:08:43.150 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.150 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.150 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.150 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.351083+0000", 2024-04-26T01:08:43.150 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0099705909999999991 2024-04-26T01:08:43.151 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.151 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.151 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.151 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.151 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.151 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:606 3.4s0 3:2d34bde3:::smithi16428812-29:head [read 0~2129920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:43.151 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.340889+0000", 2024-04-26T01:08:43.151 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 18.728321233999999, 2024-04-26T01:08:43.151 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012634348, 2024-04-26T01:08:43.151 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.151 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.151 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:43.151 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.151 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.151 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.152 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 606 2024-04-26T01:08:43.152 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.152 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.152 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.152 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.152 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340889+0000", 2024-04-26T01:08:43.152 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.152 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.152 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.152 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.152 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340889+0000", 2024-04-26T01:08:43.152 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.152 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.152 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.153 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.153 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340890+0000", 2024-04-26T01:08:43.153 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6299999999999993e-07 2024-04-26T01:08:43.153 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.153 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.153 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.153 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340896+0000", 2024-04-26T01:08:43.153 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6500000000000001e-06 2024-04-26T01:08:43.153 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.153 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.153 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.153 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340917+0000", 2024-04-26T01:08:43.153 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0769999999999999e-05 2024-04-26T01:08:43.153 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.153 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.154 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.154 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340927+0000", 2024-04-26T01:08:43.154 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0077e-05 2024-04-26T01:08:43.154 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.154 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.154 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.154 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341566+0000", 2024-04-26T01:08:43.154 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000638881 2024-04-26T01:08:43.154 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.154 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.154 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:43.154 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341677+0000", 2024-04-26T01:08:43.154 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000111595 2024-04-26T01:08:43.154 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.155 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.155 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.155 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.353524+0000", 2024-04-26T01:08:43.155 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011846412000000001 2024-04-26T01:08:43.155 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.155 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.155 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.155 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.155 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.155 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:607 3.4s0 3:2d34bde3:::smithi16428812-29:head [read 0~2129920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:43.155 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.341454+0000", 2024-04-26T01:08:43.155 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 18.727756749000001, 2024-04-26T01:08:43.155 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014935789, 2024-04-26T01:08:43.155 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.156 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.156 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:43.156 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.156 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.156 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.156 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 607 2024-04-26T01:08:43.156 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.156 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.156 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.156 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.156 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341454+0000", 2024-04-26T01:08:43.156 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.156 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.156 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.156 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.157 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341454+0000", 2024-04-26T01:08:43.157 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.157 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.157 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.157 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.157 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341455+0000", 2024-04-26T01:08:43.157 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0550000000000001e-06 2024-04-26T01:08:43.157 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.157 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.157 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.157 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341464+0000", 2024-04-26T01:08:43.157 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4350000000000003e-06 2024-04-26T01:08:43.157 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.157 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.158 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.158 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341477+0000", 2024-04-26T01:08:43.158 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2731999999999999e-05 2024-04-26T01:08:43.158 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.158 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.158 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.158 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341488+0000", 2024-04-26T01:08:43.158 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.118e-05 2024-04-26T01:08:43.158 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.158 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.158 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.158 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.343260+0000", 2024-04-26T01:08:43.158 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0017719980000000001 2024-04-26T01:08:43.158 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.158 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.159 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:43.159 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.343498+0000", 2024-04-26T01:08:43.159 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00023756600000000001 2024-04-26T01:08:43.159 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.159 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.159 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.159 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.356390+0000", 2024-04-26T01:08:43.159 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012891823 2024-04-26T01:08:43.159 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.159 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.159 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.159 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.159 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.159 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:614 3.4s0 3:2d34bde3:::smithi16428812-29:head [rollback f] snapc f=[f,d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-26T01:08:43.159 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:25.399082+0000", 2024-04-26T01:08:43.160 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.670128728999998, 2024-04-26T01:08:43.160 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010883195, 2024-04-26T01:08:43.160 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.160 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.160 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:43.160 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.160 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.160 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.160 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 614 2024-04-26T01:08:43.160 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.160 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.160 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.160 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.160 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399082+0000", 2024-04-26T01:08:43.161 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.161 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.161 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.161 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.161 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399082+0000", 2024-04-26T01:08:43.161 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.161 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.161 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.161 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.161 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399082+0000", 2024-04-26T01:08:43.161 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5000000000000003e-07 2024-04-26T01:08:43.161 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.161 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.161 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.162 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399085+0000", 2024-04-26T01:08:43.162 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.492e-06 2024-04-26T01:08:43.162 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.162 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.162 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.162 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399091+0000", 2024-04-26T01:08:43.162 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6780000000000003e-06 2024-04-26T01:08:43.162 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.162 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.162 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.162 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399096+0000", 2024-04-26T01:08:43.162 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9080000000000003e-06 2024-04-26T01:08:43.162 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.162 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.163 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.163 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.401097+0000", 2024-04-26T01:08:43.163 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0020008719999999999 2024-04-26T01:08:43.163 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.163 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.163 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:43.163 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.401190+0000", 2024-04-26T01:08:43.163 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2440000000000003e-05 2024-04-26T01:08:43.163 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.163 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.163 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.163 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.401938+0000", 2024-04-26T01:08:43.163 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00074833400000000002 2024-04-26T01:08:43.163 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.164 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.164 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.164 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.406495+0000", 2024-04-26T01:08:43.164 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0045574099999999996 2024-04-26T01:08:43.164 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.164 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.164 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:43.164 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.409843+0000", 2024-04-26T01:08:43.164 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033479970000000001 2024-04-26T01:08:43.164 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.164 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.164 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.164 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.409965+0000", 2024-04-26T01:08:43.164 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000121514 2024-04-26T01:08:43.164 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.165 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.165 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.165 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.165 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.165 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:615 3.4s0 3:2d34bde3:::smithi16428812-29:head [append 0~0] snapc f=[f,d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-26T01:08:43.165 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:25.399140+0000", 2024-04-26T01:08:43.165 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.670070481, 2024-04-26T01:08:43.165 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011121401, 2024-04-26T01:08:43.165 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.165 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.165 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:43.165 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:43.165 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:43.165 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:43.166 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 615 2024-04-26T01:08:43.166 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.166 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.166 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.166 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.166 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399140+0000", 2024-04-26T01:08:43.166 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.166 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.166 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.166 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.166 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399140+0000", 2024-04-26T01:08:43.166 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.166 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.166 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.166 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.167 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399140+0000", 2024-04-26T01:08:43.167 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4200000000000001e-07 2024-04-26T01:08:43.167 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.167 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.167 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.167 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399143+0000", 2024-04-26T01:08:43.167 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.604e-06 2024-04-26T01:08:43.167 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.167 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.167 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.167 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399149+0000", 2024-04-26T01:08:43.167 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3450000000000002e-06 2024-04-26T01:08:43.168 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.168 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.168 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.168 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.399155+0000", 2024-04-26T01:08:43.168 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9800000000000003e-06 2024-04-26T01:08:43.168 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.168 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.168 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.168 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.402924+0000", 2024-04-26T01:08:43.168 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0037682430000000001 2024-04-26T01:08:43.168 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.168 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.168 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:43.168 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.403216+0000", 2024-04-26T01:08:43.168 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00029238099999999998 2024-04-26T01:08:43.169 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.169 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.169 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.169 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.404019+0000", 2024-04-26T01:08:43.169 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00080257600000000005 2024-04-26T01:08:43.169 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.169 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.169 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.169 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.407799+0000", 2024-04-26T01:08:43.169 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0037806530000000001 2024-04-26T01:08:43.169 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.169 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.169 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:43.169 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.410124+0000", 2024-04-26T01:08:43.170 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0023246429999999999 2024-04-26T01:08:43.170 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.170 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.170 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.170 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.410261+0000", 2024-04-26T01:08:43.170 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000137534 2024-04-26T01:08:43.170 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.170 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.170 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.170 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.170 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.170 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 59/30 ECSubWriteReply(tid=56, last_complete=59'40, committed=1, applied=1))", 2024-04-26T01:08:43.170 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:25.412873+0000", 2024-04-26T01:08:43.170 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.656337483000001, 2024-04-26T01:08:43.171 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.43333044199999998, 2024-04-26T01:08:43.171 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.171 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.171 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.171 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.171 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.171 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.171 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.412873+0000", 2024-04-26T01:08:43.171 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.171 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.171 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.171 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.171 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.412873+0000", 2024-04-26T01:08:43.171 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.172 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.172 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.172 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.172 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.412874+0000", 2024-04-26T01:08:43.172 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3600000000000003e-07 2024-04-26T01:08:43.172 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.172 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.172 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.172 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.412881+0000", 2024-04-26T01:08:43.172 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3699999999999997e-06 2024-04-26T01:08:43.172 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.172 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.172 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.172 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.846041+0000", 2024-04-26T01:08:43.173 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.43316028400000001 2024-04-26T01:08:43.173 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.173 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.173 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.173 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.846075+0000", 2024-04-26T01:08:43.173 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3213000000000002e-05 2024-04-26T01:08:43.173 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.173 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.173 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.173 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.846162+0000", 2024-04-26T01:08:43.173 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7017999999999999e-05 2024-04-26T01:08:43.173 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.173 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.173 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.173 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.846203+0000", 2024-04-26T01:08:43.174 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1921000000000001e-05 2024-04-26T01:08:43.174 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.174 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.174 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.174 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.174 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.174 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:43.175 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:08:43.176 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:43.173+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:43.271 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.13 to 1 (10000) 2024-04-26T01:08:43.275 INFO:tasks.rados.rados.0.smithi164.stdout:333: done (1 left) 2024-04-26T01:08:43.275 INFO:tasks.rados.rados.0.smithi164.stdout:334: done (0 left) 2024-04-26T01:08:43.275 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 7 current snap is 23 2024-04-26T01:08:43.276 INFO:tasks.rados.rados.0.smithi164.stdout:335: seq_num 146 ranges {0=49152} 2024-04-26T01:08:43.276 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:43.272+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:43.277 INFO:tasks.rados.rados.0.smithi164.stdout:335: writing smithi16428812-7 from 0 to 49152 tid 1 2024-04-26T01:08:43.277 INFO:tasks.rados.rados.0.smithi164.stdout:336: read oid 43 snap -1 2024-04-26T01:08:43.277 INFO:tasks.rados.rados.0.smithi164.stdout:336: expect (ObjNum 104 snap 11 seq_num 104) 2024-04-26T01:08:43.277 INFO:tasks.rados.rados.0.smithi164.stdout:337: rollback oid 15 current snap is 23 2024-04-26T01:08:43.277 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 15 to 19 2024-04-26T01:08:43.278 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 33 current snap is 23 2024-04-26T01:08:43.278 INFO:tasks.rados.rados.0.smithi164.stdout:338: seq_num 147 ranges {4046848=65536} 2024-04-26T01:08:43.301 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-04-26T01:08:43.313 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:43.313 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:43.313 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-04-26T01:08:43.313 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:43.327 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:08:43.339 INFO:tasks.rados.rados.0.smithi164.stdout:338: writing smithi16428812-33 from 4046848 to 4112384 tid 1 2024-04-26T01:08:43.339 INFO:tasks.rados.rados.0.smithi164.stdout:339: snap_create 2024-04-26T01:08:43.339 INFO:tasks.rados.rados.0.smithi164.stdout:335: finishing write tid 1 to smithi16428812-7 2024-04-26T01:08:43.339 INFO:tasks.rados.rados.0.smithi164.stdout:335: finishing write tid 2 to smithi16428812-7 2024-04-26T01:08:43.340 INFO:tasks.rados.rados.0.smithi164.stdout:335: finishing write tid 3 to smithi16428812-7 2024-04-26T01:08:43.340 INFO:tasks.rados.rados.0.smithi164.stdout:335: oid 7 updating version 0 to 86 2024-04-26T01:08:43.340 INFO:tasks.rados.rados.0.smithi164.stdout:335: oid 7 updating version 86 to 87 2024-04-26T01:08:43.340 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 7 v 87 (ObjNum 146 snap 23 seq_num 146) dirty exists 2024-04-26T01:08:43.340 INFO:tasks.rados.rados.0.smithi164.stdout:335: left oid 7 (ObjNum 146 snap 23 seq_num 146) 2024-04-26T01:08:43.340 INFO:tasks.rados.rados.0.smithi164.stdout:337: finishing rollback tid 0 to smithi16428812-15 2024-04-26T01:08:43.340 INFO:tasks.rados.rados.0.smithi164.stdout:337: finishing rollback tid 1 to smithi16428812-15 2024-04-26T01:08:43.340 INFO:tasks.rados.rados.0.smithi164.stdout:337: finishing rollback tid 2 to smithi16428812-15 2024-04-26T01:08:43.340 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 15 v 63 (ObjNum 118 snap 15 seq_num 118) dirty exists 2024-04-26T01:08:43.340 INFO:tasks.rados.rados.0.smithi164.stdout:336: expect (ObjNum 104 snap 11 seq_num 104) 2024-04-26T01:08:43.377 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:43.373+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:43.433 INFO:tasks.rados.rados.0.smithi164.stdout:338: finishing write tid 1 to smithi16428812-33 2024-04-26T01:08:43.433 INFO:tasks.rados.rados.0.smithi164.stdout:338: finishing write tid 2 to smithi16428812-33 2024-04-26T01:08:43.433 INFO:tasks.rados.rados.0.smithi164.stdout:338: finishing write tid 3 to smithi16428812-33 2024-04-26T01:08:43.433 INFO:tasks.rados.rados.0.smithi164.stdout:338: oid 33 updating version 0 to 92 2024-04-26T01:08:43.433 INFO:tasks.rados.rados.0.smithi164.stdout:338: oid 33 version 92 is already newer than 91 2024-04-26T01:08:43.433 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 33 v 92 (ObjNum 147 snap 23 seq_num 147) dirty exists 2024-04-26T01:08:43.433 INFO:tasks.rados.rados.0.smithi164.stdout:338: left oid 33 (ObjNum 147 snap 23 seq_num 147) 2024-04-26T01:08:43.478 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:43.479+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:43.490 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:43.490 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:43.490 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-04-26T01:08:43.490 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-04-26T01:08:43.490 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:43.504 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:08:43.579 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:43.579+0000 7ff2871de640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 32810) UID: 0 2024-04-26T01:08:43.622 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:43.623 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:43.623 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:43.623 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:43.623 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.623 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 33/30 ECSubWrite(tid=0, reqid=client.4521.0:22, at_version=33'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:43.623 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.310594+0000", 2024-04-26T01:08:43.623 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 44.310429321999997, 2024-04-26T01:08:43.623 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035400710000000001, 2024-04-26T01:08:43.623 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.623 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.623 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.623 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.623 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.623 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.624 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310594+0000", 2024-04-26T01:08:43.624 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.624 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.624 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.624 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310594+0000", 2024-04-26T01:08:43.624 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.624 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.624 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.624 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310594+0000", 2024-04-26T01:08:43.624 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7500000000000001e-07 2024-04-26T01:08:43.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.624 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.625 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.625 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310673+0000", 2024-04-26T01:08:43.625 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8784999999999999e-05 2024-04-26T01:08:43.625 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.625 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.625 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.625 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310747+0000", 2024-04-26T01:08:43.625 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3899999999999994e-05 2024-04-26T01:08:43.625 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.625 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.625 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.625 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310763+0000", 2024-04-26T01:08:43.625 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5769000000000001e-05 2024-04-26T01:08:43.626 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.626 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.626 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.626 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310858+0000", 2024-04-26T01:08:43.626 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.5383000000000005e-05 2024-04-26T01:08:43.626 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.626 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.626 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.626 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.310871+0000", 2024-04-26T01:08:43.626 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2887e-05 2024-04-26T01:08:43.626 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.626 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.626 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.626 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314085+0000", 2024-04-26T01:08:43.627 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032136249999999999 2024-04-26T01:08:43.627 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.627 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.627 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.627 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314134+0000", 2024-04-26T01:08:43.627 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9447000000000003e-05 2024-04-26T01:08:43.627 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.627 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.627 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.627 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.627 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.627 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 32/30 ECSubWrite(tid=1, reqid=client.4521.0:25, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:43.627 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.312642+0000", 2024-04-26T01:08:43.627 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 44.308381093999998, 2024-04-26T01:08:43.628 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032273459999999999, 2024-04-26T01:08:43.628 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.628 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.628 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.628 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.628 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.628 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.628 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312642+0000", 2024-04-26T01:08:43.628 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.628 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.628 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.628 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.628 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312642+0000", 2024-04-26T01:08:43.628 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.629 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.629 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.629 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.629 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312644+0000", 2024-04-26T01:08:43.629 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2720000000000001e-06 2024-04-26T01:08:43.629 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.629 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.629 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.629 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312878+0000", 2024-04-26T01:08:43.629 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00023409 2024-04-26T01:08:43.629 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.629 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.629 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.629 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312930+0000", 2024-04-26T01:08:43.629 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2799000000000001e-05 2024-04-26T01:08:43.630 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.630 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.630 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.630 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.312944+0000", 2024-04-26T01:08:43.630 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3687e-05 2024-04-26T01:08:43.630 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.630 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.630 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.630 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.313045+0000", 2024-04-26T01:08:43.630 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000100782 2024-04-26T01:08:43.630 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.630 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.630 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.630 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.313058+0000", 2024-04-26T01:08:43.631 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2785999999999999e-05 2024-04-26T01:08:43.631 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.631 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.631 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.631 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.315782+0000", 2024-04-26T01:08:43.631 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027242439999999998 2024-04-26T01:08:43.631 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.631 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.631 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.631 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.315870+0000", 2024-04-26T01:08:43.631 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7686e-05 2024-04-26T01:08:43.631 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.631 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.631 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.632 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.632 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.632 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 38/30 ECSubWrite(tid=27, reqid=osd.2.0:27, at_version=38'21, trim_to=0'0, roll_forward_to=37'19))", 2024-04-26T01:08:43.632 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.359904+0000", 2024-04-26T01:08:43.632 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 39.261118994, 2024-04-26T01:08:43.632 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0030231030000000001, 2024-04-26T01:08:43.632 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.632 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.632 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.632 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.632 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.632 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.632 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.359904+0000", 2024-04-26T01:08:43.632 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.633 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.633 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.633 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.633 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.359904+0000", 2024-04-26T01:08:43.633 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.633 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.633 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.633 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.633 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.359905+0000", 2024-04-26T01:08:43.633 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1399999999999997e-07 2024-04-26T01:08:43.634 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.634 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.634 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.634 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.359945+0000", 2024-04-26T01:08:43.634 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0318999999999998e-05 2024-04-26T01:08:43.634 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.634 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.634 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.634 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.360030+0000", 2024-04-26T01:08:43.634 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5159000000000002e-05 2024-04-26T01:08:43.634 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.634 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.634 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.634 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.360050+0000", 2024-04-26T01:08:43.634 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9412999999999999e-05 2024-04-26T01:08:43.635 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.635 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.635 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.635 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.360211+0000", 2024-04-26T01:08:43.635 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000161652 2024-04-26T01:08:43.635 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.635 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.635 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.635 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.360242+0000", 2024-04-26T01:08:43.635 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0522999999999999e-05 2024-04-26T01:08:43.635 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.635 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.635 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.635 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.362859+0000", 2024-04-26T01:08:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0026173099999999999 2024-04-26T01:08:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.362927+0000", 2024-04-26T01:08:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8312999999999995e-05 2024-04-26T01:08:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 38/30 ECSubWrite(tid=72, reqid=client.4521.0:263, at_version=38'22, trim_to=0'0, roll_forward_to=36'21))", 2024-04-26T01:08:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.408150+0000", 2024-04-26T01:08:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 39.212873827999999, 2024-04-26T01:08:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032528909999999999, 2024-04-26T01:08:43.637 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.637 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.637 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.637 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.637 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.637 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.637 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.408150+0000", 2024-04-26T01:08:43.637 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.637 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.637 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.637 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.637 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.408150+0000", 2024-04-26T01:08:43.637 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.638 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.638 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.638 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.638 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.408150+0000", 2024-04-26T01:08:43.638 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6399999999999998e-07 2024-04-26T01:08:43.638 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.638 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.638 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.638 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.408742+0000", 2024-04-26T01:08:43.638 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00059185499999999996 2024-04-26T01:08:43.638 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.638 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.638 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.638 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.408823+0000", 2024-04-26T01:08:43.639 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0949e-05 2024-04-26T01:08:43.639 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.639 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.639 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.639 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.408841+0000", 2024-04-26T01:08:43.639 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7876e-05 2024-04-26T01:08:43.639 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.639 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.639 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.639 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.409004+0000", 2024-04-26T01:08:43.639 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016316200000000001 2024-04-26T01:08:43.639 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.639 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.639 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.639 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.409031+0000", 2024-04-26T01:08:43.640 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7160000000000001e-05 2024-04-26T01:08:43.640 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.640 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.640 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.640 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.411310+0000", 2024-04-26T01:08:43.640 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022787329999999998 2024-04-26T01:08:43.640 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.640 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.640 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.640 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.411402+0000", 2024-04-26T01:08:43.640 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2792000000000002e-05 2024-04-26T01:08:43.640 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.640 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.640 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.641 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.641 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.641 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 38/30 ECSubWrite(tid=74, reqid=client.4521.0:264, at_version=38'23, trim_to=0'0, roll_forward_to=38'22))", 2024-04-26T01:08:43.641 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:04.415518+0000", 2024-04-26T01:08:43.641 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 39.205504904000001, 2024-04-26T01:08:43.641 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032675970000000001, 2024-04-26T01:08:43.641 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.641 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.641 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.641 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.641 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.641 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.641 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.415518+0000", 2024-04-26T01:08:43.641 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.642 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.642 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.642 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.642 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.415518+0000", 2024-04-26T01:08:43.642 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.642 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.642 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.642 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.642 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.415520+0000", 2024-04-26T01:08:43.642 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6270000000000001e-06 2024-04-26T01:08:43.642 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.642 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.643 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.643 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.415852+0000", 2024-04-26T01:08:43.643 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00033187000000000001 2024-04-26T01:08:43.643 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.643 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.643 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.643 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.416299+0000", 2024-04-26T01:08:43.643 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00044752600000000001 2024-04-26T01:08:43.643 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.643 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.643 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.643 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.416325+0000", 2024-04-26T01:08:43.644 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5259999999999999e-05 2024-04-26T01:08:43.644 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.644 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.644 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.644 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.416639+0000", 2024-04-26T01:08:43.644 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00031457399999999997 2024-04-26T01:08:43.644 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.644 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.644 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.644 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.416663+0000", 2024-04-26T01:08:43.645 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3583000000000001e-05 2024-04-26T01:08:43.645 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.645 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.645 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.645 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.418708+0000", 2024-04-26T01:08:43.645 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0020455550000000001 2024-04-26T01:08:43.645 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.645 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.645 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.645 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:04.418786+0000", 2024-04-26T01:08:43.645 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7601999999999993e-05 2024-04-26T01:08:43.645 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.645 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.646 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.646 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.646 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.646 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs3 50/30 ECSubWrite(tid=214, reqid=osd.4.0:214, at_version=50'22, trim_to=0'0, roll_forward_to=45'20))", 2024-04-26T01:08:43.646 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:16.046662+0000", 2024-04-26T01:08:43.646 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 27.574361367000002, 2024-04-26T01:08:43.646 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0043439849999999999, 2024-04-26T01:08:43.646 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.646 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.646 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.646 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.647 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.647 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.647 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046662+0000", 2024-04-26T01:08:43.647 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.647 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.647 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.647 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.647 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046662+0000", 2024-04-26T01:08:43.647 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.647 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.647 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.647 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.647 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046662+0000", 2024-04-26T01:08:43.647 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6500000000000001e-07 2024-04-26T01:08:43.647 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.648 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.648 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.648 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046688+0000", 2024-04-26T01:08:43.648 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6217e-05 2024-04-26T01:08:43.648 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.648 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.648 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.648 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046803+0000", 2024-04-26T01:08:43.648 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000114837 2024-04-26T01:08:43.648 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.648 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.648 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.648 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046812+0000", 2024-04-26T01:08:43.648 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2560000000000008e-06 2024-04-26T01:08:43.648 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.649 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.649 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.649 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046879+0000", 2024-04-26T01:08:43.649 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6812999999999999e-05 2024-04-26T01:08:43.649 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.649 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.649 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.649 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.046890+0000", 2024-04-26T01:08:43.649 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0842999999999999e-05 2024-04-26T01:08:43.649 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.649 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.649 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.649 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.050916+0000", 2024-04-26T01:08:43.649 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0040258079999999996 2024-04-26T01:08:43.650 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.650 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.650 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.650 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051006+0000", 2024-04-26T01:08:43.650 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0045999999999998e-05 2024-04-26T01:08:43.650 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.650 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.650 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.650 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.650 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.650 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.6s4 50/30 ECSubWrite(tid=41, reqid=osd.0.0:41, at_version=50'24, trim_to=0'0, roll_forward_to=45'22))", 2024-04-26T01:08:43.651 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:16.051314+0000", 2024-04-26T01:08:43.651 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 27.569709678999999, 2024-04-26T01:08:43.651 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003465143, 2024-04-26T01:08:43.651 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.651 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.651 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.651 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.651 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.651 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.651 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051314+0000", 2024-04-26T01:08:43.651 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.651 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.651 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.652 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.652 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051314+0000", 2024-04-26T01:08:43.652 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.652 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.652 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.652 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.652 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051314+0000", 2024-04-26T01:08:43.652 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0800000000000001e-07 2024-04-26T01:08:43.652 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.652 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.652 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.652 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051361+0000", 2024-04-26T01:08:43.652 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6925000000000002e-05 2024-04-26T01:08:43.653 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.653 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.653 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.653 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051412+0000", 2024-04-26T01:08:43.653 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1161e-05 2024-04-26T01:08:43.653 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.653 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.653 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.653 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051421+0000", 2024-04-26T01:08:43.653 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2839999999999995e-06 2024-04-26T01:08:43.653 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.653 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.653 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.653 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051864+0000", 2024-04-26T01:08:43.654 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00044309800000000001 2024-04-26T01:08:43.654 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.654 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.654 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.654 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.051893+0000", 2024-04-26T01:08:43.654 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8782999999999999e-05 2024-04-26T01:08:43.654 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.654 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.654 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.654 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.054700+0000", 2024-04-26T01:08:43.655 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0028072549999999998 2024-04-26T01:08:43.655 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.655 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.655 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.655 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:16.054779+0000", 2024-04-26T01:08:43.655 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8429000000000004e-05 2024-04-26T01:08:43.655 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.655 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.655 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.655 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.656 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.656 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs3 55/30 ECSubWrite(tid=252, reqid=client.4521.0:548, at_version=55'28, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:43.656 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:21.083658+0000", 2024-04-26T01:08:43.656 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 22.537365285, 2024-04-26T01:08:43.656 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0087585849999999993, 2024-04-26T01:08:43.656 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.656 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.656 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.656 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.656 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.656 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.657 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.083658+0000", 2024-04-26T01:08:43.657 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.657 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.657 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.657 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.657 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.083658+0000", 2024-04-26T01:08:43.657 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.657 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.657 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.657 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.657 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.083659+0000", 2024-04-26T01:08:43.657 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3300000000000002e-07 2024-04-26T01:08:43.657 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.658 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.658 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.658 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.083943+0000", 2024-04-26T01:08:43.658 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00028484799999999998 2024-04-26T01:08:43.658 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.658 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.658 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.658 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.084081+0000", 2024-04-26T01:08:43.658 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000137818 2024-04-26T01:08:43.658 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.658 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.659 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.659 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.084108+0000", 2024-04-26T01:08:43.659 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6287000000000001e-05 2024-04-26T01:08:43.659 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.659 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.659 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.659 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.084277+0000", 2024-04-26T01:08:43.659 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016978800000000001 2024-04-26T01:08:43.659 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.660 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.660 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.660 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.084304+0000", 2024-04-26T01:08:43.660 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6967000000000001e-05 2024-04-26T01:08:43.660 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.660 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.660 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.660 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.092229+0000", 2024-04-26T01:08:43.660 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0079250229999999998 2024-04-26T01:08:43.660 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.660 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.660 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.661 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.092417+0000", 2024-04-26T01:08:43.661 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000187321 2024-04-26T01:08:43.661 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.661 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.661 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.661 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.661 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.661 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs3 55/30 ECSubWrite(tid=254, reqid=client.4521.0:549, at_version=55'29, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:43.661 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:21.087618+0000", 2024-04-26T01:08:43.661 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 22.533404953000002, 2024-04-26T01:08:43.661 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0099810469999999998, 2024-04-26T01:08:43.661 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.661 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.661 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.662 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.662 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.662 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.662 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.087618+0000", 2024-04-26T01:08:43.662 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.662 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.662 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.662 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.662 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.087618+0000", 2024-04-26T01:08:43.662 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.662 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.662 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.662 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.662 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.087620+0000", 2024-04-26T01:08:43.663 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.86e-06 2024-04-26T01:08:43.663 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.663 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.663 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.663 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.087838+0000", 2024-04-26T01:08:43.663 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000217741 2024-04-26T01:08:43.663 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.663 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.663 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.663 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.087932+0000", 2024-04-26T01:08:43.663 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4392e-05 2024-04-26T01:08:43.663 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.663 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.663 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.664 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.087960+0000", 2024-04-26T01:08:43.664 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7376999999999999e-05 2024-04-26T01:08:43.664 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.664 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.664 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.664 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.092546+0000", 2024-04-26T01:08:43.664 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0045858519999999996 2024-04-26T01:08:43.664 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.664 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.664 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.664 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.092570+0000", 2024-04-26T01:08:43.664 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3920000000000001e-05 2024-04-26T01:08:43.664 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.664 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.665 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.665 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.097506+0000", 2024-04-26T01:08:43.665 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.004936686 2024-04-26T01:08:43.665 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.665 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.665 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.665 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.097599+0000", 2024-04-26T01:08:43.665 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3219000000000003e-05 2024-04-26T01:08:43.665 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.665 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.665 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.665 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.665 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.665 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.es6 55/44 ECSubWrite(tid=55, reqid=osd.2.0:55, at_version=55'44, trim_to=0'0, roll_forward_to=52'42))", 2024-04-26T01:08:43.665 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:21.094514+0000", 2024-04-26T01:08:43.666 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 22.526509629, 2024-04-26T01:08:43.666 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035312939999999999, 2024-04-26T01:08:43.666 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.666 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.666 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.666 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.666 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.666 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.666 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094514+0000", 2024-04-26T01:08:43.666 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.666 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.666 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.666 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.666 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094514+0000", 2024-04-26T01:08:43.667 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.667 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.667 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.667 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.667 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094514+0000", 2024-04-26T01:08:43.667 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.37e-07 2024-04-26T01:08:43.667 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.667 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.667 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.667 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094571+0000", 2024-04-26T01:08:43.667 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6693000000000001e-05 2024-04-26T01:08:43.667 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.667 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.667 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.668 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094688+0000", 2024-04-26T01:08:43.668 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011739300000000001 2024-04-26T01:08:43.668 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.668 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.668 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.668 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094712+0000", 2024-04-26T01:08:43.668 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3108999999999998e-05 2024-04-26T01:08:43.668 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.668 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.668 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.668 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094824+0000", 2024-04-26T01:08:43.668 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011225 2024-04-26T01:08:43.668 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.668 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.669 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.669 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.094838+0000", 2024-04-26T01:08:43.669 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4361e-05 2024-04-26T01:08:43.669 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.669 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.669 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.669 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.097990+0000", 2024-04-26T01:08:43.669 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0031517070000000001 2024-04-26T01:08:43.669 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.669 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.669 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.669 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.098045+0000", 2024-04-26T01:08:43.669 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5143999999999999e-05 2024-04-26T01:08:43.670 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.670 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.670 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.670 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.670 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.670 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs3 55/30 ECSubWrite(tid=256, reqid=client.4521.0:550, at_version=55'30, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:43.670 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:21.102027+0000", 2024-04-26T01:08:43.670 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 22.518996709, 2024-04-26T01:08:43.670 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0037147669999999999, 2024-04-26T01:08:43.670 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.670 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.670 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.670 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.670 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.671 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.671 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102027+0000", 2024-04-26T01:08:43.671 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.671 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.671 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.671 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.671 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102027+0000", 2024-04-26T01:08:43.671 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.671 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.671 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.671 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.671 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102028+0000", 2024-04-26T01:08:43.671 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.703e-06 2024-04-26T01:08:43.671 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.672 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.672 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.672 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102149+0000", 2024-04-26T01:08:43.672 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000121149 2024-04-26T01:08:43.672 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.672 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.672 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.672 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102279+0000", 2024-04-26T01:08:43.672 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012956299999999999 2024-04-26T01:08:43.672 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.672 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.672 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.672 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102305+0000", 2024-04-26T01:08:43.672 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.552e-05 2024-04-26T01:08:43.673 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.673 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.673 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.673 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102458+0000", 2024-04-26T01:08:43.673 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015376100000000001 2024-04-26T01:08:43.673 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.673 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.673 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.673 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.102485+0000", 2024-04-26T01:08:43.673 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6529999999999998e-05 2024-04-26T01:08:43.674 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.674 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.674 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.674 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.105647+0000", 2024-04-26T01:08:43.674 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003162045 2024-04-26T01:08:43.674 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.674 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.674 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.674 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:21.105741+0000", 2024-04-26T01:08:43.674 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4495999999999998e-05 2024-04-26T01:08:43.674 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.674 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.674 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.674 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.675 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.675 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs3 59/30 ECSubWrite(tid=331, reqid=client.4521.0:616, at_version=59'47, trim_to=0'0, roll_forward_to=59'45))", 2024-04-26T01:08:43.675 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:25.463449+0000", 2024-04-26T01:08:43.675 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 18.157573929000002, 2024-04-26T01:08:43.675 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0038682870000000002, 2024-04-26T01:08:43.675 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.675 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.675 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.675 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.675 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.676 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.676 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.463449+0000", 2024-04-26T01:08:43.676 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.676 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.676 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.676 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.676 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.463449+0000", 2024-04-26T01:08:43.676 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.676 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.676 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.676 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.676 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.463452+0000", 2024-04-26T01:08:43.676 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5600000000000001e-06 2024-04-26T01:08:43.676 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.677 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.677 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.677 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.463935+0000", 2024-04-26T01:08:43.677 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00048299299999999997 2024-04-26T01:08:43.677 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.677 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.677 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.677 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.464046+0000", 2024-04-26T01:08:43.677 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011133000000000001 2024-04-26T01:08:43.677 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.678 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.678 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.678 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.464071+0000", 2024-04-26T01:08:43.678 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4502999999999999e-05 2024-04-26T01:08:43.678 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.678 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.678 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.678 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.464240+0000", 2024-04-26T01:08:43.678 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016889800000000001 2024-04-26T01:08:43.678 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.678 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.678 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.678 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.464268+0000", 2024-04-26T01:08:43.679 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7886999999999999e-05 2024-04-26T01:08:43.679 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.679 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.679 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.679 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.467212+0000", 2024-04-26T01:08:43.679 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0029448640000000002 2024-04-26T01:08:43.679 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.679 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.679 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.679 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.467318+0000", 2024-04-26T01:08:43.679 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000105252 2024-04-26T01:08:43.680 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.680 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.680 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.680 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.680 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.680 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.6s4 60/30 ECSubWrite(tid=59, reqid=client.4521.0:641, at_version=60'39, trim_to=0'0, roll_forward_to=58'37))", 2024-04-26T01:08:43.680 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:26.174498+0000", 2024-04-26T01:08:43.680 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.446525736000002, 2024-04-26T01:08:43.680 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003094865, 2024-04-26T01:08:43.680 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.680 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.680 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.681 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.681 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.681 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.681 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174498+0000", 2024-04-26T01:08:43.681 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.681 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.681 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.682 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.682 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174498+0000", 2024-04-26T01:08:43.682 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.682 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.682 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.682 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.682 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174498+0000", 2024-04-26T01:08:43.682 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5300000000000004e-07 2024-04-26T01:08:43.682 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.682 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.683 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.683 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174537+0000", 2024-04-26T01:08:43.683 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8435e-05 2024-04-26T01:08:43.683 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.683 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.683 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.683 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174618+0000", 2024-04-26T01:08:43.683 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1435999999999998e-05 2024-04-26T01:08:43.683 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.683 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.683 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.684 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174632+0000", 2024-04-26T01:08:43.684 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3919e-05 2024-04-26T01:08:43.684 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.684 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.684 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.684 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174719+0000", 2024-04-26T01:08:43.684 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7492999999999997e-05 2024-04-26T01:08:43.684 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.684 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.684 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.684 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.174734+0000", 2024-04-26T01:08:43.685 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.455e-05 2024-04-26T01:08:43.685 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.685 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.685 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.685 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.177532+0000", 2024-04-26T01:08:43.685 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027984720000000002 2024-04-26T01:08:43.685 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.685 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.685 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.685 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.177592+0000", 2024-04-26T01:08:43.685 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0007000000000003e-05 2024-04-26T01:08:43.685 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.685 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.686 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.686 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.686 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.686 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs3 60/30 ECSubWrite(tid=351, reqid=osd.4.0:351, at_version=60'54, trim_to=0'0, roll_forward_to=59'50))", 2024-04-26T01:08:43.686 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:26.178228+0000", 2024-04-26T01:08:43.686 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.442795812, 2024-04-26T01:08:43.686 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003424889, 2024-04-26T01:08:43.686 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.686 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.686 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.686 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.686 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.686 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.687 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.178228+0000", 2024-04-26T01:08:43.687 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.687 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.687 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.687 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.687 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.178228+0000", 2024-04-26T01:08:43.687 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.687 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.687 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.687 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.687 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.178228+0000", 2024-04-26T01:08:43.688 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9399999999999995e-07 2024-04-26T01:08:43.688 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.688 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.688 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.688 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.178254+0000", 2024-04-26T01:08:43.688 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5741e-05 2024-04-26T01:08:43.688 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.688 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.688 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.688 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.178376+0000", 2024-04-26T01:08:43.688 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000122063 2024-04-26T01:08:43.688 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.689 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.689 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.689 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.178388+0000", 2024-04-26T01:08:43.689 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1905999999999999e-05 2024-04-26T01:08:43.689 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.689 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.689 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.689 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.179887+0000", 2024-04-26T01:08:43.689 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0014988029999999999 2024-04-26T01:08:43.689 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.689 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.689 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.689 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.179900+0000", 2024-04-26T01:08:43.690 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3042e-05 2024-04-26T01:08:43.690 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.690 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.690 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.690 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.181602+0000", 2024-04-26T01:08:43.690 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0017023190000000001 2024-04-26T01:08:43.690 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.690 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.690 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.690 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.181652+0000", 2024-04-26T01:08:43.690 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.0521e-05 2024-04-26T01:08:43.690 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.691 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.691 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.691 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.691 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.691 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 65/44 ECSubWrite(tid=62, reqid=client.4521.0:704, at_version=65'70, trim_to=0'0, roll_forward_to=61'68))", 2024-04-26T01:08:43.691 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:31.078935+0000", 2024-04-26T01:08:43.691 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.542088297999999, 2024-04-26T01:08:43.691 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033798180000000001, 2024-04-26T01:08:43.691 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.691 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.691 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.691 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.691 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.691 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.692 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:31.078935+0000", 2024-04-26T01:08:43.692 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.692 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.692 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.692 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.692 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:31.078935+0000", 2024-04-26T01:08:43.692 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.692 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.692 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.692 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.692 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:31.078938+0000", 2024-04-26T01:08:43.692 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8049999999999998e-06 2024-04-26T01:08:43.692 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.692 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.693 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.693 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:31.079245+0000", 2024-04-26T01:08:43.693 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00030703499999999999 2024-04-26T01:08:43.693 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.693 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.693 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.693 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:31.079303+0000", 2024-04-26T01:08:43.693 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8556e-05 2024-04-26T01:08:43.693 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.693 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.693 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.693 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:31.079317+0000", 2024-04-26T01:08:43.693 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3545e-05 2024-04-26T01:08:43.693 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.694 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.694 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.694 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:31.079408+0000", 2024-04-26T01:08:43.694 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1225999999999997e-05 2024-04-26T01:08:43.694 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.694 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.694 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.694 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:31.079421+0000", 2024-04-26T01:08:43.694 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2604e-05 2024-04-26T01:08:43.694 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.694 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.694 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.694 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:31.082249+0000", 2024-04-26T01:08:43.694 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0028281019999999999 2024-04-26T01:08:43.695 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.695 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.695 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.695 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:31.082315+0000", 2024-04-26T01:08:43.695 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5945000000000006e-05 2024-04-26T01:08:43.695 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.695 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.695 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.695 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.695 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.695 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 66/44 ECSubWrite(tid=64, reqid=osd.9.0:64, at_version=66'72, trim_to=0'0, roll_forward_to=65'70))", 2024-04-26T01:08:43.695 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:32.856858+0000", 2024-04-26T01:08:43.695 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 10.764164961000001, 2024-04-26T01:08:43.695 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033222109999999998, 2024-04-26T01:08:43.696 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.696 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.696 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.696 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.696 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.696 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.696 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:32.856858+0000", 2024-04-26T01:08:43.696 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.696 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.696 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.696 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.696 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:32.856858+0000", 2024-04-26T01:08:43.696 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.697 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.697 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.697 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.697 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:32.856859+0000", 2024-04-26T01:08:43.697 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4000000000000002e-07 2024-04-26T01:08:43.697 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.697 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.697 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.697 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:32.856895+0000", 2024-04-26T01:08:43.697 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6538999999999997e-05 2024-04-26T01:08:43.697 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.697 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.697 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.697 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:32.856968+0000", 2024-04-26T01:08:43.697 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3046000000000005e-05 2024-04-26T01:08:43.698 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.698 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.698 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.698 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:32.856985+0000", 2024-04-26T01:08:43.698 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6371999999999998e-05 2024-04-26T01:08:43.698 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.698 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.698 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.698 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:32.857102+0000", 2024-04-26T01:08:43.698 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000117323 2024-04-26T01:08:43.698 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.698 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.698 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.699 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:32.857120+0000", 2024-04-26T01:08:43.699 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7482999999999998e-05 2024-04-26T01:08:43.699 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.699 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.699 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.699 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:32.860129+0000", 2024-04-26T01:08:43.699 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0030094620000000001 2024-04-26T01:08:43.699 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.699 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.699 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.699 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:32.860181+0000", 2024-04-26T01:08:43.699 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1545999999999997e-05 2024-04-26T01:08:43.699 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.699 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.699 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.700 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.700 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.700 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 71/44 ECSubWrite(tid=72, reqid=client.4521.0:729, at_version=71'79, trim_to=0'0, roll_forward_to=70'78))", 2024-04-26T01:08:43.700 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.116002+0000", 2024-04-26T01:08:43.700 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 6.505021298, 2024-04-26T01:08:43.700 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00315351, 2024-04-26T01:08:43.700 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.700 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.700 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.700 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.700 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.700 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.700 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.116002+0000", 2024-04-26T01:08:43.700 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.701 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.701 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.701 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.701 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.116002+0000", 2024-04-26T01:08:43.701 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.701 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.701 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.701 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.701 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.116003+0000", 2024-04-26T01:08:43.701 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.367e-06 2024-04-26T01:08:43.701 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.701 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.701 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.701 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.116967+0000", 2024-04-26T01:08:43.702 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00096360499999999997 2024-04-26T01:08:43.702 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.702 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.702 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.702 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.117208+0000", 2024-04-26T01:08:43.702 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00024096400000000001 2024-04-26T01:08:43.702 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.702 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.702 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.702 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.117221+0000", 2024-04-26T01:08:43.702 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2768999999999999e-05 2024-04-26T01:08:43.702 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.702 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.702 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.703 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.117443+0000", 2024-04-26T01:08:43.703 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00022203199999999999 2024-04-26T01:08:43.703 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.703 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.703 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.703 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.117459+0000", 2024-04-26T01:08:43.703 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5909e-05 2024-04-26T01:08:43.703 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.703 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.703 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.703 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.119105+0000", 2024-04-26T01:08:43.703 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0016463700000000001 2024-04-26T01:08:43.703 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.703 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.704 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.704 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.119156+0000", 2024-04-26T01:08:43.704 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.0494000000000001e-05 2024-04-26T01:08:43.704 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.704 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.704 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.704 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.704 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.704 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 71/44 ECSubWrite(tid=73, reqid=osd.9.0:73, at_version=71'81, trim_to=0'0, roll_forward_to=70'78))", 2024-04-26T01:08:43.704 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.116973+0000", 2024-04-26T01:08:43.704 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 6.5040501449999999, 2024-04-26T01:08:43.704 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0041564829999999999, 2024-04-26T01:08:43.704 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.704 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.705 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.705 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.705 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.705 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.705 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.116973+0000", 2024-04-26T01:08:43.705 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.705 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.705 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.705 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.705 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.116973+0000", 2024-04-26T01:08:43.705 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.705 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.705 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.705 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.706 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.116974+0000", 2024-04-26T01:08:43.706 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5400000000000001e-07 2024-04-26T01:08:43.706 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.706 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.706 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.706 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.116993+0000", 2024-04-26T01:08:43.706 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9626000000000001e-05 2024-04-26T01:08:43.706 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.706 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.706 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.706 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.117371+0000", 2024-04-26T01:08:43.707 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00037766500000000001 2024-04-26T01:08:43.707 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.707 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.707 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.707 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.117388+0000", 2024-04-26T01:08:43.707 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.664e-05 2024-04-26T01:08:43.707 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.707 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.707 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.707 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.118298+0000", 2024-04-26T01:08:43.707 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00091069299999999996 2024-04-26T01:08:43.707 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.707 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.708 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.708 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.118312+0000", 2024-04-26T01:08:43.708 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3349999999999999e-05 2024-04-26T01:08:43.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.708 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.708 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.708 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.121077+0000", 2024-04-26T01:08:43.708 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027646860000000001 2024-04-26T01:08:43.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.708 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.708 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.708 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.121130+0000", 2024-04-26T01:08:43.709 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3168999999999999e-05 2024-04-26T01:08:43.709 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.709 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.709 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.709 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.709 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.709 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs5 71/44 ECSubWrite(tid=74, reqid=client.4521.0:730, at_version=71'82, trim_to=0'0, roll_forward_to=70'78))", 2024-04-26T01:08:43.709 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.116996+0000", 2024-04-26T01:08:43.709 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 6.5040273879999999, 2024-04-26T01:08:43.709 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0048385559999999999, 2024-04-26T01:08:43.709 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.709 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.710 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.710 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.710 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.710 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.710 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.116996+0000", 2024-04-26T01:08:43.710 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.710 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.710 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.710 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.710 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.116996+0000", 2024-04-26T01:08:43.710 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.711 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.711 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.711 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.711 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.116996+0000", 2024-04-26T01:08:43.711 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.2899999999999999e-07 2024-04-26T01:08:43.711 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.711 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.711 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.711 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.117157+0000", 2024-04-26T01:08:43.711 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016076999999999999 2024-04-26T01:08:43.711 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.711 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.712 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.712 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.117474+0000", 2024-04-26T01:08:43.712 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00031744299999999999 2024-04-26T01:08:43.712 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.712 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.712 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.712 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.117484+0000", 2024-04-26T01:08:43.712 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6560000000000004e-06 2024-04-26T01:08:43.712 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.712 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.712 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.713 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.119334+0000", 2024-04-26T01:08:43.713 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0018501329999999999 2024-04-26T01:08:43.713 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.713 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.713 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.713 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.119347+0000", 2024-04-26T01:08:43.713 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2466999999999999e-05 2024-04-26T01:08:43.713 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.713 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.713 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.713 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.121801+0000", 2024-04-26T01:08:43.713 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024541720000000001 2024-04-26T01:08:43.714 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.714 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.714 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.714 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.121835+0000", 2024-04-26T01:08:43.714 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3586e-05 2024-04-26T01:08:43.714 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.714 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.714 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.714 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.714 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.714 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.6s4 71/30 ECSubWrite(tid=83, reqid=client.4521.0:743, at_version=71'55, trim_to=0'0, roll_forward_to=67'54))", 2024-04-26T01:08:43.715 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.169918+0000", 2024-04-26T01:08:43.715 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 6.4511056700000005, 2024-04-26T01:08:43.715 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003032398, 2024-04-26T01:08:43.715 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:43.715 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:43.715 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:43.715 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:43.715 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.715 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:43.716 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.169918+0000", 2024-04-26T01:08:43.716 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.716 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.716 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.716 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:43.716 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.169918+0000", 2024-04-26T01:08:43.716 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:43.716 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.716 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.716 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:43.716 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.169918+0000", 2024-04-26T01:08:43.717 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3200000000000005e-07 2024-04-26T01:08:43.717 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.717 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.717 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:43.717 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.170079+0000", 2024-04-26T01:08:43.717 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016130199999999999 2024-04-26T01:08:43.717 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.717 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.717 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:43.717 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.170277+0000", 2024-04-26T01:08:43.717 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019740799999999999 2024-04-26T01:08:43.718 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.718 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.718 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:43.718 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.170541+0000", 2024-04-26T01:08:43.718 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000264367 2024-04-26T01:08:43.718 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.718 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.718 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:43.718 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.170788+0000", 2024-04-26T01:08:43.718 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00024678599999999998 2024-04-26T01:08:43.718 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.718 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.718 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:43.719 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.170815+0000", 2024-04-26T01:08:43.719 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6460000000000001e-05 2024-04-26T01:08:43.719 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.719 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.719 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:43.719 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.172898+0000", 2024-04-26T01:08:43.719 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0020833179999999998 2024-04-26T01:08:43.719 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:43.719 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:43.719 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:43.719 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.172950+0000", 2024-04-26T01:08:43.720 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2225000000000002e-05 2024-04-26T01:08:43.720 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.720 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.720 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.720 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:43.720 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:43.720 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:43.722 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:08:43.722 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:43.719+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:43.823 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:43.821+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:43.858 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: "num_ops": 0 2024-04-26T01:08:43.858 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:43.872 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:08:43.923 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:43.923+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:44.024 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:44.021+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:44.043 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:08:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:08:44.044 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:44.058 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:08:44.078 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-26T01:08:44.088 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:08:44.125 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:44.123+0000 7fba11fca640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25263) UID: 0 2024-04-26T01:08:44.226 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:44.227+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:44.231 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:08:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:08:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:08:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.2s5 32/30 ECSubWrite(tid=157, reqid=client.4521.0:13, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.308314+0000", 2024-04-26T01:08:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 44.921736213000003, 2024-04-26T01:08:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011288362999999999, 2024-04-26T01:08:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308314+0000", 2024-04-26T01:08:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308314+0000", 2024-04-26T01:08:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308315+0000", 2024-04-26T01:08:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.6599999999999995e-07 2024-04-26T01:08:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308467+0000", 2024-04-26T01:08:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000152249 2024-04-26T01:08:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308602+0000", 2024-04-26T01:08:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000135048 2024-04-26T01:08:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308653+0000", 2024-04-26T01:08:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0864e-05 2024-04-26T01:08:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308834+0000", 2024-04-26T01:08:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000180466 2024-04-26T01:08:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308857+0000", 2024-04-26T01:08:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2904e-05 2024-04-26T01:08:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.319506+0000", 2024-04-26T01:08:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010649141000000001 2024-04-26T01:08:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.319603+0000", 2024-04-26T01:08:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6924999999999998e-05 2024-04-26T01:08:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 32/30 ECSubWrite(tid=0, reqid=client.4521.0:16, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 44.919517894000002, 2024-04-26T01:08:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012119314000000001, 2024-04-26T01:08:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.7000000000000004e-07 2024-04-26T01:08:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310733+0000", 2024-04-26T01:08:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019985599999999999 2024-04-26T01:08:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310792+0000", 2024-04-26T01:08:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.8894000000000002e-05 2024-04-26T01:08:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310816+0000", 2024-04-26T01:08:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4094e-05 2024-04-26T01:08:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311022+0000", 2024-04-26T01:08:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00020608599999999999 2024-04-26T01:08:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311044+0000", 2024-04-26T01:08:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1889999999999999e-05 2024-04-26T01:08:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322407+0000", 2024-04-26T01:08:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011362778 2024-04-26T01:08:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322652+0000", 2024-04-26T01:08:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00024504600000000001 2024-04-26T01:08:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.es2 33/30 ECSubWrite(tid=0, reqid=client.4521.0:22, at_version=33'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.311344+0000", 2024-04-26T01:08:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 44.918706380000003, 2024-04-26T01:08:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010979113, 2024-04-26T01:08:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311344+0000", 2024-04-26T01:08:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311344+0000", 2024-04-26T01:08:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311351+0000", 2024-04-26T01:08:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1040000000000001e-06 2024-04-26T01:08:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311496+0000", 2024-04-26T01:08:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000144852 2024-04-26T01:08:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311639+0000", 2024-04-26T01:08:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014267800000000001 2024-04-26T01:08:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311663+0000", 2024-04-26T01:08:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4090000000000001e-05 2024-04-26T01:08:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311836+0000", 2024-04-26T01:08:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000173349 2024-04-26T01:08:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311859+0000", 2024-04-26T01:08:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2504e-05 2024-04-26T01:08:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322193+0000", 2024-04-26T01:08:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010334157 2024-04-26T01:08:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322323+0000", 2024-04-26T01:08:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013037899999999999 2024-04-26T01:08:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.8s3 32/30 ECSubWrite(tid=1, reqid=client.4521.0:8, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.312999+0000", 2024-04-26T01:08:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 44.917051917999999, 2024-04-26T01:08:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0094850960000000002, 2024-04-26T01:08:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312999+0000", 2024-04-26T01:08:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312999+0000", 2024-04-26T01:08:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313001+0000", 2024-04-26T01:08:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.21e-06 2024-04-26T01:08:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313076+0000", 2024-04-26T01:08:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.5525000000000006e-05 2024-04-26T01:08:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313194+0000", 2024-04-26T01:08:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000118004 2024-04-26T01:08:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313238+0000", 2024-04-26T01:08:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3523999999999999e-05 2024-04-26T01:08:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313570+0000", 2024-04-26T01:08:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033178100000000001 2024-04-26T01:08:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313595+0000", 2024-04-26T01:08:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5181000000000002e-05 2024-04-26T01:08:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322175+0000", 2024-04-26T01:08:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0085803070000000006 2024-04-26T01:08:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322484+0000", 2024-04-26T01:08:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000308564 2024-04-26T01:08:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.es2 33/30 ECSubWrite(tid=1, reqid=client.4521.0:23, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.313493+0000", 2024-04-26T01:08:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 44.916557885000003, 2024-04-26T01:08:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0089452720000000006, 2024-04-26T01:08:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313493+0000", 2024-04-26T01:08:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313493+0000", 2024-04-26T01:08:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313494+0000", 2024-04-26T01:08:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5039999999999999e-06 2024-04-26T01:08:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313593+0000", 2024-04-26T01:08:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.9355e-05 2024-04-26T01:08:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313756+0000", 2024-04-26T01:08:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000162762 2024-04-26T01:08:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313781+0000", 2024-04-26T01:08:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4893e-05 2024-04-26T01:08:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313944+0000", 2024-04-26T01:08:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016297600000000001 2024-04-26T01:08:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313968+0000", 2024-04-26T01:08:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3986999999999999e-05 2024-04-26T01:08:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322340+0000", 2024-04-26T01:08:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0083720840000000001 2024-04-26T01:08:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322438+0000", 2024-04-26T01:08:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7711e-05 2024-04-26T01:08:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 50/30 ECSubWrite(tid=29, reqid=osd.8.0:29, at_version=50'18, trim_to=0'0, roll_forward_to=47'16))", 2024-04-26T01:08:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.040723+0000", 2024-04-26T01:08:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 28.189327644999999, 2024-04-26T01:08:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012574701000000001, 2024-04-26T01:08:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040723+0000", 2024-04-26T01:08:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040723+0000", 2024-04-26T01:08:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040724+0000", 2024-04-26T01:08:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.37e-07 2024-04-26T01:08:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040778+0000", 2024-04-26T01:08:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.4453000000000002e-05 2024-04-26T01:08:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040860+0000", 2024-04-26T01:08:44.256 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.2060999999999999e-05 2024-04-26T01:08:44.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.256 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.256 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.256 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040883+0000", 2024-04-26T01:08:44.256 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2994e-05 2024-04-26T01:08:44.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.256 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.256 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.256 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.041058+0000", 2024-04-26T01:08:44.256 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017496999999999999 2024-04-26T01:08:44.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.256 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.256 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.257 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.041082+0000", 2024-04-26T01:08:44.257 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.393e-05 2024-04-26T01:08:44.257 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.257 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.257 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053181+0000", 2024-04-26T01:08:44.257 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012099085000000001 2024-04-26T01:08:44.257 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.257 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.257 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053298+0000", 2024-04-26T01:08:44.257 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000116471 2024-04-26T01:08:44.257 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.257 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 50/30 ECSubWrite(tid=214, reqid=osd.4.0:214, at_version=50'22, trim_to=0'0, roll_forward_to=45'20))", 2024-04-26T01:08:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.047877+0000", 2024-04-26T01:08:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 28.182172981000001, 2024-04-26T01:08:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010720746, 2024-04-26T01:08:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047877+0000", 2024-04-26T01:08:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047877+0000", 2024-04-26T01:08:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047878+0000", 2024-04-26T01:08:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.6699999999999999e-07 2024-04-26T01:08:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047930+0000", 2024-04-26T01:08:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.1669999999999998e-05 2024-04-26T01:08:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048154+0000", 2024-04-26T01:08:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022397399999999999 2024-04-26T01:08:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048176+0000", 2024-04-26T01:08:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2734e-05 2024-04-26T01:08:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048452+0000", 2024-04-26T01:08:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00027581799999999999 2024-04-26T01:08:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048478+0000", 2024-04-26T01:08:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5723000000000001e-05 2024-04-26T01:08:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058506+0000", 2024-04-26T01:08:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010027635999999999 2024-04-26T01:08:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058598+0000", 2024-04-26T01:08:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2724000000000002e-05 2024-04-26T01:08:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.6s2 50/30 ECSubWrite(tid=41, reqid=osd.0.0:41, at_version=50'24, trim_to=0'0, roll_forward_to=45'22))", 2024-04-26T01:08:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.052765+0000", 2024-04-26T01:08:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 28.177285237, 2024-04-26T01:08:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0080835920000000006, 2024-04-26T01:08:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052765+0000", 2024-04-26T01:08:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052765+0000", 2024-04-26T01:08:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052766+0000", 2024-04-26T01:08:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1999999999999999e-07 2024-04-26T01:08:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052820+0000", 2024-04-26T01:08:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.4441000000000003e-05 2024-04-26T01:08:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052916+0000", 2024-04-26T01:08:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5597e-05 2024-04-26T01:08:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052940+0000", 2024-04-26T01:08:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3626e-05 2024-04-26T01:08:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053100+0000", 2024-04-26T01:08:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015994800000000001 2024-04-26T01:08:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053126+0000", 2024-04-26T01:08:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6393e-05 2024-04-26T01:08:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060748+0000", 2024-04-26T01:08:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.007622523 2024-04-26T01:08:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060849+0000", 2024-04-26T01:08:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000100344 2024-04-26T01:08:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 55/30 ECSubWrite(tid=254, reqid=client.4521.0:549, at_version=55'29, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:21.088071+0000", 2024-04-26T01:08:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 23.141979042999999, 2024-04-26T01:08:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.023607655000000002, 2024-04-26T01:08:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.088071+0000", 2024-04-26T01:08:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.088071+0000", 2024-04-26T01:08:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.088074+0000", 2024-04-26T01:08:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.097e-06 2024-04-26T01:08:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.088360+0000", 2024-04-26T01:08:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000286261 2024-04-26T01:08:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.103040+0000", 2024-04-26T01:08:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.014680442 2024-04-26T01:08:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.103049+0000", 2024-04-26T01:08:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.251e-06 2024-04-26T01:08:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.103185+0000", 2024-04-26T01:08:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013595400000000001 2024-04-26T01:08:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.103212+0000", 2024-04-26T01:08:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6517e-05 2024-04-26T01:08:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.111554+0000", 2024-04-26T01:08:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.008342426 2024-04-26T01:08:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.111679+0000", 2024-04-26T01:08:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000124707 2024-04-26T01:08:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 55/30 ECSubWrite(tid=256, reqid=client.4521.0:550, at_version=55'30, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:21.102883+0000", 2024-04-26T01:08:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 23.127167190000002, 2024-04-26T01:08:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0094720329999999995, 2024-04-26T01:08:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.102883+0000", 2024-04-26T01:08:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.102883+0000", 2024-04-26T01:08:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.102884+0000", 2024-04-26T01:08:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8000000000000005e-07 2024-04-26T01:08:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.103016+0000", 2024-04-26T01:08:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013183699999999999 2024-04-26T01:08:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105250+0000", 2024-04-26T01:08:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002234307 2024-04-26T01:08:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105272+0000", 2024-04-26T01:08:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1886e-05 2024-04-26T01:08:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105352+0000", 2024-04-26T01:08:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.0383999999999995e-05 2024-04-26T01:08:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105364+0000", 2024-04-26T01:08:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1331999999999999e-05 2024-04-26T01:08:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.112272+0000", 2024-04-26T01:08:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0069084970000000004 2024-04-26T01:08:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.112355+0000", 2024-04-26T01:08:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.3010000000000007e-05 2024-04-26T01:08:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 55/30 ECSubWrite(tid=257, reqid=client.4521.0:551, at_version=55'31, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:21.105099+0000", 2024-04-26T01:08:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 23.124951094, 2024-04-26T01:08:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0073293309999999997, 2024-04-26T01:08:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105099+0000", 2024-04-26T01:08:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105099+0000", 2024-04-26T01:08:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105101+0000", 2024-04-26T01:08:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.75e-06 2024-04-26T01:08:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105182+0000", 2024-04-26T01:08:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.0533999999999998e-05 2024-04-26T01:08:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107687+0000", 2024-04-26T01:08:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0025055659999999999 2024-04-26T01:08:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107716+0000", 2024-04-26T01:08:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8750000000000001e-05 2024-04-26T01:08:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107877+0000", 2024-04-26T01:08:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016068300000000001 2024-04-26T01:08:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107902+0000", 2024-04-26T01:08:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.561e-05 2024-04-26T01:08:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.112364+0000", 2024-04-26T01:08:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0044621039999999997 2024-04-26T01:08:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.112429+0000", 2024-04-26T01:08:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.4333999999999998e-05 2024-04-26T01:08:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 55/30 ECSubWrite(tid=259, reqid=client.4521.0:552, at_version=55'32, trim_to=0'0, roll_forward_to=55'28))", 2024-04-26T01:08:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:21.107592+0000", 2024-04-26T01:08:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 23.122458728000002, 2024-04-26T01:08:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0078481520000000006, 2024-04-26T01:08:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107592+0000", 2024-04-26T01:08:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107592+0000", 2024-04-26T01:08:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107594+0000", 2024-04-26T01:08:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.764e-06 2024-04-26T01:08:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.107640+0000", 2024-04-26T01:08:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.6866000000000002e-05 2024-04-26T01:08:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.113932+0000", 2024-04-26T01:08:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0062914210000000002 2024-04-26T01:08:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.113942+0000", 2024-04-26T01:08:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0627e-05 2024-04-26T01:08:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.114006+0000", 2024-04-26T01:08:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.3448999999999995e-05 2024-04-26T01:08:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.114016+0000", 2024-04-26T01:08:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.9550000000000004e-06 2024-04-26T01:08:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.115404+0000", 2024-04-26T01:08:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0013883980000000001 2024-04-26T01:08:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.115440+0000", 2024-04-26T01:08:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.5672e-05 2024-04-26T01:08:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpRead(3.bs2 55/30 ECSubRead(tid=262, to_read={3:d5691d7a:::smithi16428812-40:head=0,438272,0}, subchunks={3:d5691d7a:::smithi16428812-40:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:21.113903+0000", 2024-04-26T01:08:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 23.116147138999999, 2024-04-26T01:08:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0071693190000000004, 2024-04-26T01:08:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.113903+0000", 2024-04-26T01:08:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.113903+0000", 2024-04-26T01:08:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.113904+0000", 2024-04-26T01:08:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8900000000000001e-07 2024-04-26T01:08:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.113912+0000", 2024-04-26T01:08:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.9540000000000004e-06 2024-04-26T01:08:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.120147+0000", 2024-04-26T01:08:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0062352040000000003 2024-04-26T01:08:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.120175+0000", 2024-04-26T01:08:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.83e-05 2024-04-26T01:08:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.120260+0000", 2024-04-26T01:08:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.4646000000000002e-05 2024-04-26T01:08:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.121073+0000", 2024-04-26T01:08:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00081292599999999995 2024-04-26T01:08:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s3 58/30 ECSubWrite(tid=62, reqid=osd.14.0:62, at_version=58'47, trim_to=0'0, roll_forward_to=56'45))", 2024-04-26T01:08:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:24.166131+0000", 2024-04-26T01:08:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 20.063919468000002, 2024-04-26T01:08:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0076169009999999997, 2024-04-26T01:08:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166131+0000", 2024-04-26T01:08:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166131+0000", 2024-04-26T01:08:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166132+0000", 2024-04-26T01:08:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.9299999999999998e-07 2024-04-26T01:08:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166185+0000", 2024-04-26T01:08:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.3712999999999999e-05 2024-04-26T01:08:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166289+0000", 2024-04-26T01:08:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000103391 2024-04-26T01:08:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166311+0000", 2024-04-26T01:08:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2359999999999999e-05 2024-04-26T01:08:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166470+0000", 2024-04-26T01:08:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000158543 2024-04-26T01:08:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.166499+0000", 2024-04-26T01:08:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9074e-05 2024-04-26T01:08:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.173647+0000", 2024-04-26T01:08:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0071487269999999997 2024-04-26T01:08:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.173748+0000", 2024-04-26T01:08:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0001005 2024-04-26T01:08:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 59/30 ECSubWrite(tid=320, reqid=osd.4.0:320, at_version=59'43, trim_to=0'0, roll_forward_to=58'41))", 2024-04-26T01:08:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:25.158834+0000", 2024-04-26T01:08:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 19.071216806999999, 2024-04-26T01:08:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0071564360000000004, 2024-04-26T01:08:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.158834+0000", 2024-04-26T01:08:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.158834+0000", 2024-04-26T01:08:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.158834+0000", 2024-04-26T01:08:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.2999999999999999e-07 2024-04-26T01:08:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.158893+0000", 2024-04-26T01:08:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.808e-05 2024-04-26T01:08:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.159012+0000", 2024-04-26T01:08:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000119657 2024-04-26T01:08:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.159039+0000", 2024-04-26T01:08:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7189999999999999e-05 2024-04-26T01:08:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.159216+0000", 2024-04-26T01:08:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017637600000000001 2024-04-26T01:08:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.159242+0000", 2024-04-26T01:08:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6239999999999999e-05 2024-04-26T01:08:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.165863+0000", 2024-04-26T01:08:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0066210829999999998 2024-04-26T01:08:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.165990+0000", 2024-04-26T01:08:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012698000000000001 2024-04-26T01:08:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s3 59/30 ECSubWrite(tid=66, reqid=osd.14.0:66, at_version=59'51, trim_to=0'0, roll_forward_to=58'49))", 2024-04-26T01:08:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:25.171370+0000", 2024-04-26T01:08:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 19.058680520999999, 2024-04-26T01:08:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00777277, 2024-04-26T01:08:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.171370+0000", 2024-04-26T01:08:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.171370+0000", 2024-04-26T01:08:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.171371+0000", 2024-04-26T01:08:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.31e-06 2024-04-26T01:08:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.171423+0000", 2024-04-26T01:08:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.1323000000000003e-05 2024-04-26T01:08:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.171527+0000", 2024-04-26T01:08:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000103984 2024-04-26T01:08:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.171551+0000", 2024-04-26T01:08:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4470000000000001e-05 2024-04-26T01:08:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.173214+0000", 2024-04-26T01:08:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0016628050000000001 2024-04-26T01:08:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.173261+0000", 2024-04-26T01:08:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.7273e-05 2024-04-26T01:08:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.179091+0000", 2024-04-26T01:08:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0058294380000000002 2024-04-26T01:08:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.179143+0000", 2024-04-26T01:08:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.2166999999999997e-05 2024-04-26T01:08:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 59/30 ECSubWrite(tid=53, reqid=client.4521.0:613, at_version=59'38, trim_to=0'0, roll_forward_to=59'36))", 2024-04-26T01:08:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:25.401515+0000", 2024-04-26T01:08:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 18.828535001999999, 2024-04-26T01:08:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0070293689999999997, 2024-04-26T01:08:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401515+0000", 2024-04-26T01:08:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401515+0000", 2024-04-26T01:08:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401517+0000", 2024-04-26T01:08:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.917e-06 2024-04-26T01:08:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401559+0000", 2024-04-26T01:08:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.1180000000000002e-05 2024-04-26T01:08:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401650+0000", 2024-04-26T01:08:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.1663000000000007e-05 2024-04-26T01:08:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401666+0000", 2024-04-26T01:08:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5583999999999999e-05 2024-04-26T01:08:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401848+0000", 2024-04-26T01:08:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00018238999999999999 2024-04-26T01:08:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.401880+0000", 2024-04-26T01:08:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.1446999999999999e-05 2024-04-26T01:08:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.408444+0000", 2024-04-26T01:08:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0065639050000000001 2024-04-26T01:08:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.408545+0000", 2024-04-26T01:08:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000101283 2024-04-26T01:08:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 60/30 ECSubWrite(tid=349, reqid=osd.4.0:349, at_version=60'52, trim_to=0'0, roll_forward_to=59'50))", 2024-04-26T01:08:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:26.179457+0000", 2024-04-26T01:08:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 18.050593215999999, 2024-04-26T01:08:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0076187240000000003, 2024-04-26T01:08:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179457+0000", 2024-04-26T01:08:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179457+0000", 2024-04-26T01:08:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179458+0000", 2024-04-26T01:08:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3700000000000001e-07 2024-04-26T01:08:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179498+0000", 2024-04-26T01:08:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.0726999999999998e-05 2024-04-26T01:08:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179634+0000", 2024-04-26T01:08:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013533299999999999 2024-04-26T01:08:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179655+0000", 2024-04-26T01:08:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1710000000000001e-05 2024-04-26T01:08:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179855+0000", 2024-04-26T01:08:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019919 2024-04-26T01:08:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179882+0000", 2024-04-26T01:08:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7250000000000002e-05 2024-04-26T01:08:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.186989+0000", 2024-04-26T01:08:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0071068729999999997 2024-04-26T01:08:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.187076+0000", 2024-04-26T01:08:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.7204000000000001e-05 2024-04-26T01:08:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 60/30 ECSubWrite(tid=351, reqid=osd.4.0:351, at_version=60'54, trim_to=0'0, roll_forward_to=59'50))", 2024-04-26T01:08:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:26.179502+0000", 2024-04-26T01:08:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 18.050548446000001, 2024-04-26T01:08:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.009331499, 2024-04-26T01:08:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179502+0000", 2024-04-26T01:08:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179502+0000", 2024-04-26T01:08:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179502+0000", 2024-04-26T01:08:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.7e-07 2024-04-26T01:08:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179532+0000", 2024-04-26T01:08:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9977000000000001e-05 2024-04-26T01:08:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179726+0000", 2024-04-26T01:08:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019319300000000001 2024-04-26T01:08:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179745+0000", 2024-04-26T01:08:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9167000000000001e-05 2024-04-26T01:08:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.183638+0000", 2024-04-26T01:08:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0038929580000000002 2024-04-26T01:08:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.183663+0000", 2024-04-26T01:08:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5673999999999998e-05 2024-04-26T01:08:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.188734+0000", 2024-04-26T01:08:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0050702530000000003 2024-04-26T01:08:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.188834+0000", 2024-04-26T01:08:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.9807000000000002e-05 2024-04-26T01:08:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 60/30 ECSubWrite(tid=58, reqid=osd.8.0:58, at_version=60'44, trim_to=0'0, roll_forward_to=59'40))", 2024-04-26T01:08:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:26.179976+0000", 2024-04-26T01:08:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 18.050074192, 2024-04-26T01:08:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0070517189999999997, 2024-04-26T01:08:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179976+0000", 2024-04-26T01:08:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179976+0000", 2024-04-26T01:08:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179978+0000", 2024-04-26T01:08:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6670000000000001e-06 2024-04-26T01:08:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.180025+0000", 2024-04-26T01:08:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.7227e-05 2024-04-26T01:08:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.180097+0000", 2024-04-26T01:08:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1366999999999996e-05 2024-04-26T01:08:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.180114+0000", 2024-04-26T01:08:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7506000000000001e-05 2024-04-26T01:08:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.180245+0000", 2024-04-26T01:08:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013061400000000001 2024-04-26T01:08:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.180259+0000", 2024-04-26T01:08:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.472e-05 2024-04-26T01:08:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.186934+0000", 2024-04-26T01:08:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.006674681 2024-04-26T01:08:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.187028+0000", 2024-04-26T01:08:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.3937000000000005e-05 2024-04-26T01:08:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:44.319 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:44.319 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:44.320 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:44.320 INFO:tasks.rados.rados.0.smithi164.stdout:335: done (4 left) 2024-04-26T01:08:44.320 INFO:tasks.rados.rados.0.smithi164.stdout:336: done (3 left) 2024-04-26T01:08:44.320 INFO:tasks.rados.rados.0.smithi164.stdout:337: done (2 left) 2024-04-26T01:08:44.320 INFO:tasks.rados.rados.0.smithi164.stdout:338: done (1 left) 2024-04-26T01:08:44.320 INFO:tasks.rados.rados.0.smithi164.stdout:339: done (0 left) 2024-04-26T01:08:44.320 INFO:tasks.rados.rados.0.smithi164.stdout:340: snap_create 2024-04-26T01:08:44.321 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:44.322 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:08:44.322 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.14 to 1 (10000) 2024-04-26T01:08:44.323 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-04-26T01:08:44.323 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:44.323 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:44.345 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:44.346 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:44.371 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-04-26T01:08:44.371 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:08:44.371 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:44.369+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:44.393 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:44.393 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:44.420 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:44.421 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:44.427 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:44.425+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:44.445 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:44.446 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:44.454 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:44.454 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:44.470 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:08:44.476 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:44.476 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:44.497 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:44.497 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:44.519 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:44.520 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:44.544 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:44.545 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:44.555 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:44.553+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:44.572 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:08:44.573 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:08:44.598 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:44.598 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:44.627 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:44.627 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:44.629 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:44.627+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:44.638 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:44.638 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:44.652 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:08:44.657 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:44.658 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:44.658 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:44.656+0000 7fac17fff640 0 -- 172.21.15.8:0/3206457543 >> [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] conn(0x7fac18224150 msgr2=0x7fac18226540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-04-26T01:08:44.730 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:44.728+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:44.823 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:44.824 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:44.831 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:44.831+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:44.839 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:08:44.893 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:44.894 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:44.932 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:44.931+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:44.966 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:44.966 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:44.980 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:08:45.032 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:45.028+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:45.104 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:45.104 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:45.118 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:08:45.166 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:45.161+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:45.234 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:45.235+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:45.243 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:45.243 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:45.258 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:08:45.286 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-04-26T01:08:45.287 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.15 to 1 (10000) 2024-04-26T01:08:45.292 INFO:tasks.rados.rados.0.smithi164.stdout:340: done (0 left) 2024-04-26T01:08:45.293 INFO:tasks.rados.rados.0.smithi164.stdout:341: setattr oid 48 current snap is 25 2024-04-26T01:08:45.299 INFO:tasks.rados.rados.0.smithi164.stdout:342: snap_create 2024-04-26T01:08:45.310 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 48 v 65 (ObjNum 131 snap 19 seq_num 131) dirty exists 2024-04-26T01:08:45.323 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-26T01:08:45.324 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:08:45.334 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:45.331+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:45.432 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:45.432 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:45.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:45.432+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:45.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:08:45.536 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:45.535+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:45.571 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:45.571 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:45.586 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:08:45.638 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:45.636+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:45.692 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:08:45.693 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-26T01:08:45.706 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:45.706 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:45.711 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-04-26T01:08:45.290058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":116982304,"num_objects":65,"num_object_clones":17,"num_object_copies":445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":4,"num_read":562,"num_read_kb":163056,"num_write":950,"num_write_kb":163876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":17498112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1054,"ondisk_log_size":1054,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":655408,"kb_used_data":204300,"kb_used_omap":23,"kb_used_meta":450856,"kb_avail":1414922192,"statfs":{"total":1449551462400,"available":1448880324608,"internally_reserved":0,"allocated":209203200,"data_stored":245993382,"data_compressed":2509816,"data_compressed_allocated":3461120,"data_compressed_original":7782400,"omap_allocated":23926,"internal_metadata":461677194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14778368,"num_objects":13,"num_object_clones":11,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":37,"num_read_kb":47201,"num_write":74,"num_write_kb":14864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.245344"},"pg_stats":[{"pgid":"3.f","version":"76'104","reported_seq":280,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.275456+0000","last_change":"2024-04-26T01:08:42.059546+0000","last_active":"2024-04-26T01:08:43.275456+0000","last_peered":"2024-04-26T01:08:43.275456+0000","last_clean":"2024-04-26T01:08:43.275456+0000","last_became_active":"2024-04-26T01:08:11.056866+0000","last_became_peered":"2024-04-26T01:08:11.056866+0000","last_unstale":"2024-04-26T01:08:43.275456+0000","last_undegraded":"2024-04-26T01:08:43.275456+0000","last_fullsized":"2024-04-26T01:08:43.275456+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:19.091538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028016920000000002,"stat_sum":{"num_bytes":14286848,"num_objects":6,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":17588,"num_write":86,"num_write_kb":17328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3129344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,14,13,8,10,0],"acting":[4,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.e","version":"74'56","reported_seq":322,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.390297+0000","last_change":"2024-04-26T01:08:42.066150+0000","last_active":"2024-04-26T01:08:43.390297+0000","last_peered":"2024-04-26T01:08:43.390297+0000","last_clean":"2024-04-26T01:08:43.390297+0000","last_became_active":"2024-04-26T01:08:11.054010+0000","last_became_peered":"2024-04-26T01:08:11.054010+0000","last_unstale":"2024-04-26T01:08:43.390297+0000","last_undegraded":"2024-04-26T01:08:43.390297+0000","last_fullsized":"2024-04-26T01:08:43.390297+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:25.162677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021227889999999999,"stat_sum":{"num_bytes":5111808,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":16042,"num_write":27,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3112960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,7,8,6,12],"acting":[2,9,1,7,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.d","version":"72'29","reported_seq":275,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.290655+0000","last_change":"2024-04-26T01:08:42.052060+0000","last_active":"2024-04-26T01:08:44.290655+0000","last_peered":"2024-04-26T01:08:44.290655+0000","last_clean":"2024-04-26T01:08:44.290655+0000","last_became_active":"2024-04-26T01:07:57.589029+0000","last_became_peered":"2024-04-26T01:07:57.589029+0000","last_unstale":"2024-04-26T01:08:44.290655+0000","last_undegraded":"2024-04-26T01:08:44.290655+0000","last_fullsized":"2024-04-26T01:08:44.290655+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:09.119203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000565436,"stat_sum":{"num_bytes":1523712,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":775,"num_write":15,"num_write_kb":1648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.c","version":"77'92","reported_seq":267,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.419381+0000","last_change":"2024-04-26T01:08:42.110191+0000","last_active":"2024-04-26T01:08:43.419381+0000","last_peered":"2024-04-26T01:08:43.419381+0000","last_clean":"2024-04-26T01:08:43.419381+0000","last_became_active":"2024-04-26T01:08:11.076105+0000","last_became_peered":"2024-04-26T01:08:11.076105+0000","last_unstale":"2024-04-26T01:08:43.419381+0000","last_undegraded":"2024-04-26T01:08:43.419381+0000","last_fullsized":"2024-04-26T01:08:43.419381+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:02.907098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013337010000000001,"stat_sum":{"num_bytes":15106048,"num_objects":7,"num_object_clones":2,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":45,"num_read_kb":23107,"num_write":86,"num_write_kb":11888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4849664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0,10,6,12,5],"acting":[9,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.b","version":"71'63","reported_seq":312,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.275532+0000","last_change":"2024-04-26T01:08:42.059965+0000","last_active":"2024-04-26T01:08:43.275532+0000","last_peered":"2024-04-26T01:08:43.275532+0000","last_clean":"2024-04-26T01:08:43.275532+0000","last_became_active":"2024-04-26T01:07:57.596133+0000","last_became_peered":"2024-04-26T01:07:57.596133+0000","last_unstale":"2024-04-26T01:08:43.275532+0000","last_undegraded":"2024-04-26T01:08:43.275532+0000","last_fullsized":"2024-04-26T01:08:43.275532+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:25.237963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029999380000000002,"stat_sum":{"num_bytes":8192000,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":18874,"num_write":33,"num_write_kb":11424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.a","version":"74'14","reported_seq":267,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.861040+0000","last_change":"2024-04-26T01:08:42.290581+0000","last_active":"2024-04-26T01:08:44.861040+0000","last_peered":"2024-04-26T01:08:44.861040+0000","last_clean":"2024-04-26T01:08:44.861040+0000","last_became_active":"2024-04-26T01:08:11.250401+0000","last_became_peered":"2024-04-26T01:08:11.250401+0000","last_unstale":"2024-04-26T01:08:44.861040+0000","last_undegraded":"2024-04-26T01:08:44.861040+0000","last_fullsized":"2024-04-26T01:08:44.861040+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:11.820230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001080372,"stat_sum":{"num_bytes":1490944,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":4371,"num_write":7,"num_write_kb":1456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,10],"acting":[3,7,1,9,14,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.9","version":"0'0","reported_seq":234,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.284994+0000","last_change":"2024-04-26T01:08:42.059468+0000","last_active":"2024-04-26T01:08:43.284994+0000","last_peered":"2024-04-26T01:08:43.284994+0000","last_clean":"2024-04-26T01:08:43.284994+0000","last_became_active":"2024-04-26T01:07:57.584830+0000","last_became_peered":"2024-04-26T01:07:57.584830+0000","last_unstale":"2024-04-26T01:08:43.284994+0000","last_undegraded":"2024-04-26T01:08:43.284994+0000","last_fullsized":"2024-04-26T01:08:43.284994+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:11.015264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010981879999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.8","version":"74'44","reported_seq":301,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.317446+0000","last_change":"2024-04-26T01:08:42.109999+0000","last_active":"2024-04-26T01:08:44.317446+0000","last_peered":"2024-04-26T01:08:44.317446+0000","last_clean":"2024-04-26T01:08:44.317446+0000","last_became_active":"2024-04-26T01:07:57.586870+0000","last_became_peered":"2024-04-26T01:07:57.586870+0000","last_unstale":"2024-04-26T01:08:44.317446+0000","last_undegraded":"2024-04-26T01:08:44.317446+0000","last_fullsized":"2024-04-26T01:08:44.317446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:21.490320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023345589999999999,"stat_sum":{"num_bytes":4816896,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":30,"num_read_kb":9202,"num_write":26,"num_write_kb":10400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"1.0","version":"27'157","reported_seq":313,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.859975+0000","last_change":"2024-04-26T01:07:46.409257+0000","last_active":"2024-04-26T01:08:44.859975+0000","last_peered":"2024-04-26T01:08:44.859975+0000","last_clean":"2024-04-26T01:08:44.859975+0000","last_became_active":"2024-04-26T01:07:46.407301+0000","last_became_peered":"2024-04-26T01:07:46.407301+0000","last_unstale":"2024-04-26T01:08:44.859975+0000","last_undegraded":"2024-04-26T01:08:44.859975+0000","last_fullsized":"2024-04-26T01:08:44.859975+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_clean_scrub_stamp":"2024-04-26T01:07:45.031420+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.371437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":114,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.317355+0000","last_change":"2024-04-26T01:07:48.145129+0000","last_active":"2024-04-26T01:08:44.317355+0000","last_peered":"2024-04-26T01:08:44.317355+0000","last_clean":"2024-04-26T01:08:44.317355+0000","last_became_active":"2024-04-26T01:07:48.144313+0000","last_became_peered":"2024-04-26T01:07:48.144313+0000","last_unstale":"2024-04-26T01:08:44.317355+0000","last_undegraded":"2024-04-26T01:08:44.317355+0000","last_fullsized":"2024-04-26T01:08:44.317355+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:03.023694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"73'37","reported_seq":307,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.860201+0000","last_change":"2024-04-26T01:08:42.065755+0000","last_active":"2024-04-26T01:08:44.860201+0000","last_peered":"2024-04-26T01:08:44.860201+0000","last_clean":"2024-04-26T01:08:44.860201+0000","last_became_active":"2024-04-26T01:08:11.059109+0000","last_became_peered":"2024-04-26T01:08:11.059109+0000","last_unstale":"2024-04-26T01:08:44.860201+0000","last_undegraded":"2024-04-26T01:08:44.860201+0000","last_fullsized":"2024-04-26T01:08:44.860201+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:19.067602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015211230000000001,"stat_sum":{"num_bytes":4866048,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":1832,"num_write":44,"num_write_kb":6208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,14],"acting":[13,2,3,6,15,1,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":113,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.860691+0000","last_change":"2024-04-26T01:07:48.149275+0000","last_active":"2024-04-26T01:08:44.860691+0000","last_peered":"2024-04-26T01:08:44.860691+0000","last_clean":"2024-04-26T01:08:44.860691+0000","last_became_active":"2024-04-26T01:07:48.148395+0000","last_became_peered":"2024-04-26T01:07:48.148395+0000","last_unstale":"2024-04-26T01:08:44.860691+0000","last_undegraded":"2024-04-26T01:08:44.860691+0000","last_fullsized":"2024-04-26T01:08:44.860691+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:49.139938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"76'95","reported_seq":373,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.275391+0000","last_change":"2024-04-26T01:08:42.061538+0000","last_active":"2024-04-26T01:08:43.275391+0000","last_peered":"2024-04-26T01:08:43.275391+0000","last_clean":"2024-04-26T01:08:43.275391+0000","last_became_active":"2024-04-26T01:08:11.061821+0000","last_became_peered":"2024-04-26T01:08:11.061821+0000","last_unstale":"2024-04-26T01:08:43.275391+0000","last_undegraded":"2024-04-26T01:08:43.275391+0000","last_fullsized":"2024-04-26T01:08:43.275391+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:04.412956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048083900000000001,"stat_sum":{"num_bytes":9191424,"num_objects":6,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":34,"num_read_kb":8847,"num_write":92,"num_write_kb":14720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3522560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,9,5,10,2,13],"acting":[4,7,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":113,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.859803+0000","last_change":"2024-04-26T01:07:48.146870+0000","last_active":"2024-04-26T01:08:44.859803+0000","last_peered":"2024-04-26T01:08:44.859803+0000","last_clean":"2024-04-26T01:08:44.859803+0000","last_became_active":"2024-04-26T01:07:48.145991+0000","last_became_peered":"2024-04-26T01:07:48.145991+0000","last_unstale":"2024-04-26T01:08:44.859803+0000","last_undegraded":"2024-04-26T01:08:44.859803+0000","last_fullsized":"2024-04-26T01:08:44.859803+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:13.523679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"74'77","reported_seq":324,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.285630+0000","last_change":"2024-04-26T01:08:42.059479+0000","last_active":"2024-04-26T01:08:43.285630+0000","last_peered":"2024-04-26T01:08:43.285630+0000","last_clean":"2024-04-26T01:08:43.285630+0000","last_became_active":"2024-04-26T01:07:57.587743+0000","last_became_peered":"2024-04-26T01:07:57.587743+0000","last_unstale":"2024-04-26T01:08:43.285630+0000","last_undegraded":"2024-04-26T01:08:43.285630+0000","last_fullsized":"2024-04-26T01:08:43.285630+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:06.308832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001306572,"stat_sum":{"num_bytes":11862016,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":3324,"num_write":62,"num_write_kb":20144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":113,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.860989+0000","last_change":"2024-04-26T01:07:48.147756+0000","last_active":"2024-04-26T01:08:44.860989+0000","last_peered":"2024-04-26T01:08:44.860989+0000","last_clean":"2024-04-26T01:08:44.860989+0000","last_became_active":"2024-04-26T01:07:48.147003+0000","last_became_peered":"2024-04-26T01:07:48.147003+0000","last_unstale":"2024-04-26T01:08:44.860989+0000","last_undegraded":"2024-04-26T01:08:44.860989+0000","last_fullsized":"2024-04-26T01:08:44.860989+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:05.491411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"72'20","reported_seq":264,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.859987+0000","last_change":"2024-04-26T01:08:42.065635+0000","last_active":"2024-04-26T01:08:44.859987+0000","last_peered":"2024-04-26T01:08:44.859987+0000","last_clean":"2024-04-26T01:08:44.859987+0000","last_became_active":"2024-04-26T01:07:57.310765+0000","last_became_peered":"2024-04-26T01:07:57.310765+0000","last_unstale":"2024-04-26T01:08:44.859987+0000","last_undegraded":"2024-04-26T01:08:44.859987+0000","last_fullsized":"2024-04-26T01:08:44.859987+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.890017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001596313,"stat_sum":{"num_bytes":3506176,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":4325,"num_write":35,"num_write_kb":4960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.5","version":"74'50","reported_seq":291,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.300439+0000","last_change":"2024-04-26T01:08:42.101123+0000","last_active":"2024-04-26T01:08:43.300439+0000","last_peered":"2024-04-26T01:08:43.300439+0000","last_clean":"2024-04-26T01:08:43.300439+0000","last_became_active":"2024-04-26T01:07:57.588984+0000","last_became_peered":"2024-04-26T01:07:57.588984+0000","last_unstale":"2024-04-26T01:08:43.300439+0000","last_undegraded":"2024-04-26T01:08:43.300439+0000","last_fullsized":"2024-04-26T01:08:43.300439+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:02.184107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083330600000000002,"stat_sum":{"num_bytes":4587520,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":20,"num_read_kb":17724,"num_write":25,"num_write_kb":10784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":111,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.275406+0000","last_change":"2024-04-26T01:07:48.465954+0000","last_active":"2024-04-26T01:08:43.275406+0000","last_peered":"2024-04-26T01:08:43.275406+0000","last_clean":"2024-04-26T01:08:43.275406+0000","last_became_active":"2024-04-26T01:07:48.465232+0000","last_became_peered":"2024-04-26T01:07:48.465232+0000","last_unstale":"2024-04-26T01:08:43.275406+0000","last_undegraded":"2024-04-26T01:08:43.275406+0000","last_fullsized":"2024-04-26T01:08:43.275406+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:49.270720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"71'66","reported_seq":309,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.286580+0000","last_change":"2024-04-26T01:08:42.062747+0000","last_active":"2024-04-26T01:08:43.286580+0000","last_peered":"2024-04-26T01:08:43.286580+0000","last_clean":"2024-04-26T01:08:43.286580+0000","last_became_active":"2024-04-26T01:07:57.578181+0000","last_became_peered":"2024-04-26T01:07:57.578181+0000","last_unstale":"2024-04-26T01:08:43.286580+0000","last_undegraded":"2024-04-26T01:08:43.286580+0000","last_fullsized":"2024-04-26T01:08:43.286580+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:13.423235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068971699999999996,"stat_sum":{"num_bytes":6389760,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":6899,"num_write":70,"num_write_kb":6752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":112,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.284534+0000","last_change":"2024-04-26T01:07:48.145645+0000","last_active":"2024-04-26T01:08:43.284534+0000","last_peered":"2024-04-26T01:08:43.284534+0000","last_clean":"2024-04-26T01:08:43.284534+0000","last_became_active":"2024-04-26T01:07:48.145376+0000","last_became_peered":"2024-04-26T01:07:48.145376+0000","last_unstale":"2024-04-26T01:08:43.284534+0000","last_undegraded":"2024-04-26T01:08:43.284534+0000","last_fullsized":"2024-04-26T01:08:43.284534+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.310599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"77'87","reported_seq":385,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.289248+0000","last_change":"2024-04-26T01:08:42.061999+0000","last_active":"2024-04-26T01:08:43.289248+0000","last_peered":"2024-04-26T01:08:43.289248+0000","last_clean":"2024-04-26T01:08:43.289248+0000","last_became_active":"2024-04-26T01:08:11.064770+0000","last_became_peered":"2024-04-26T01:08:11.064770+0000","last_unstale":"2024-04-26T01:08:43.289248+0000","last_undegraded":"2024-04-26T01:08:43.289248+0000","last_fullsized":"2024-04-26T01:08:43.289248+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:09.006645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049855719999999997,"stat_sum":{"num_bytes":13926400,"num_objects":8,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":76,"num_read_kb":20738,"num_write":80,"num_write_kb":21920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,14],"acting":[4,6,9,0,8,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":111,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.287174+0000","last_change":"2024-04-26T01:07:48.145909+0000","last_active":"2024-04-26T01:08:43.287174+0000","last_peered":"2024-04-26T01:08:43.287174+0000","last_clean":"2024-04-26T01:08:43.287174+0000","last_became_active":"2024-04-26T01:07:48.145512+0000","last_became_peered":"2024-04-26T01:07:48.145512+0000","last_unstale":"2024-04-26T01:08:43.287174+0000","last_undegraded":"2024-04-26T01:08:43.287174+0000","last_fullsized":"2024-04-26T01:08:43.287174+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:16.344494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"77'63","reported_seq":313,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.297905+0000","last_change":"2024-04-26T01:08:42.064126+0000","last_active":"2024-04-26T01:08:43.297905+0000","last_peered":"2024-04-26T01:08:43.297905+0000","last_clean":"2024-04-26T01:08:43.297905+0000","last_became_active":"2024-04-26T01:07:57.597503+0000","last_became_peered":"2024-04-26T01:07:57.597503+0000","last_unstale":"2024-04-26T01:08:43.297905+0000","last_undegraded":"2024-04-26T01:08:43.297905+0000","last_fullsized":"2024-04-26T01:08:43.297905+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:17.591570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060259500000000004,"stat_sum":{"num_bytes":11534336,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":9290,"num_write":59,"num_write_kb":13104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":110,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.282760+0000","last_change":"2024-04-26T01:07:48.149017+0000","last_active":"2024-04-26T01:08:43.282760+0000","last_peered":"2024-04-26T01:08:43.282760+0000","last_clean":"2024-04-26T01:08:43.282760+0000","last_became_active":"2024-04-26T01:07:48.148023+0000","last_became_peered":"2024-04-26T01:07:48.148023+0000","last_unstale":"2024-04-26T01:08:43.282760+0000","last_undegraded":"2024-04-26T01:08:43.282760+0000","last_fullsized":"2024-04-26T01:08:43.282760+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:07.554062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":116391936,"num_objects":63,"num_object_clones":17,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":4,"num_read":426,"num_read_kb":162938,"num_write":747,"num_write_kb":160416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":17498112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200904704,"data_stored":242757632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1802,"internal_metadata":0},"log_size":897,"ondisk_log_size":897,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":23216,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":21,"seq":90194313232,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38900,"kb_used_data":11880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332940,"statfs":{"total":96636764160,"available":96596930560,"internally_reserved":0,"allocated":12165120,"data_stored":15144447,"data_compressed":168835,"data_compressed_allocated":229376,"data_compressed_original":458752,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,3,4,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280526,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45368,"kb_used_data":14188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326472,"statfs":{"total":96636764160,"available":96590307328,"internally_reserved":0,"allocated":14528512,"data_stored":17596322,"data_compressed":177378,"data_compressed_allocated":266240,"data_compressed_original":962560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280528,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45324,"kb_used_data":14140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326516,"statfs":{"total":96636764160,"available":96590352384,"internally_reserved":0,"allocated":14479360,"data_stored":17323519,"data_compressed":168835,"data_compressed_allocated":229376,"data_compressed_original":458752,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,8,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313230,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54820,"kb_used_data":19480,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94317020,"statfs":{"total":96636764160,"available":96580628480,"internally_reserved":0,"allocated":19947520,"data_stored":23723394,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313231,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46752,"kb_used_data":15572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325088,"statfs":{"total":96636764160,"available":96588890112,"internally_reserved":0,"allocated":15945728,"data_stored":18595746,"data_compressed":177378,"data_compressed_allocated":266240,"data_compressed_original":962560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313232,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42088,"kb_used_data":10908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329752,"statfs":{"total":96636764160,"available":96593666048,"internally_reserved":0,"allocated":11169792,"data_stored":13282690,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313230,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49972,"kb_used_data":18784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321868,"statfs":{"total":96636764160,"available":96585592832,"internally_reserved":0,"allocated":19234816,"data_stored":20725122,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313230,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48128,"kb_used_data":16948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323712,"statfs":{"total":96636764160,"available":96587481088,"internally_reserved":0,"allocated":17354752,"data_stored":20376962,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":21,"seq":90194313230,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41764,"kb_used_data":14736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330076,"statfs":{"total":96636764160,"available":96593997824,"internally_reserved":0,"allocated":15089664,"data_stored":18234754,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313231,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42656,"kb_used_data":15628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329184,"statfs":{"total":96636764160,"available":96593084416,"internally_reserved":0,"allocated":16003072,"data_stored":20123010,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,5,6,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313230,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36444,"kb_used_data":9420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335396,"statfs":{"total":96636764160,"available":96599445504,"internally_reserved":0,"allocated":9646080,"data_stored":11474281,"data_compressed":162730,"data_compressed_allocated":221184,"data_compressed_original":442368,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":0,"seq":257698037765,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313232,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40780,"kb_used_data":9592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331060,"statfs":{"total":96636764160,"available":96595005440,"internally_reserved":0,"allocated":9822208,"data_stored":11269481,"data_compressed":162730,"data_compressed_allocated":221184,"data_compressed_original":442368,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[5,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313231,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42320,"kb_used_data":15292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329520,"statfs":{"total":96636764160,"available":96593428480,"internally_reserved":0,"allocated":15659008,"data_stored":16665986,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280528,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45284,"kb_used_data":14104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326556,"statfs":{"total":96636764160,"available":96590393344,"internally_reserved":0,"allocated":14442496,"data_stored":16731522,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280527,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34808,"kb_used_data":3628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337032,"statfs":{"total":96636764160,"available":96601120768,"internally_reserved":0,"allocated":3715072,"data_stored":4726146,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,7,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19070976,"data_stored":23093248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13565952,"data_stored":16101376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15007744,"data_stored":17436672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2838528,"data_stored":4096000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18358272,"data_stored":20094976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":13590528,"data_stored":16683008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16478208,"data_stored":19746816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11276288,"data_stored":14503936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14213120,"data_stored":17604608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":15126528,"data_stored":19492864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":9363456,"data_stored":11436032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4374528,"data_stored":4374528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1802,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":8974336,"data_stored":12967936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":13590528,"data_stored":16437248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":14782464,"data_stored":16035840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":10293248,"data_stored":12652544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:08:45.713 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:08:45.720 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:08:45.738 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:45.739+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:45.839 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:45.836+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:45.852 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:45.852 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:45.866 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:08:45.940 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:45.939+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:45.986 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:45.986 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:46.000 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:08:46.047 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:46.047+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:46.067 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:08:46.068 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-26T01:08:46.087 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-04-26T01:08:46.000976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":116982304,"num_objects":65,"num_object_clones":17,"num_object_copies":445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":4,"num_read":562,"num_read_kb":163056,"num_write":950,"num_write_kb":163876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":17498112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1054,"ondisk_log_size":1054,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":655408,"kb_used_data":204300,"kb_used_omap":23,"kb_used_meta":450856,"kb_avail":1414922192,"statfs":{"total":1449551462400,"available":1448880324608,"internally_reserved":0,"allocated":209203200,"data_stored":245993382,"data_compressed":2509816,"data_compressed_allocated":3461120,"data_compressed_original":7782400,"omap_allocated":23926,"internal_metadata":461677194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14778368,"num_objects":13,"num_object_clones":11,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":37,"num_read_kb":47201,"num_write":74,"num_write_kb":14864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001478"},"pg_stats":[{"pgid":"3.f","version":"76'104","reported_seq":280,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.275456+0000","last_change":"2024-04-26T01:08:42.059546+0000","last_active":"2024-04-26T01:08:43.275456+0000","last_peered":"2024-04-26T01:08:43.275456+0000","last_clean":"2024-04-26T01:08:43.275456+0000","last_became_active":"2024-04-26T01:08:11.056866+0000","last_became_peered":"2024-04-26T01:08:11.056866+0000","last_unstale":"2024-04-26T01:08:43.275456+0000","last_undegraded":"2024-04-26T01:08:43.275456+0000","last_fullsized":"2024-04-26T01:08:43.275456+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:19.091538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028016920000000002,"stat_sum":{"num_bytes":14286848,"num_objects":6,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":17588,"num_write":86,"num_write_kb":17328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3129344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,14,13,8,10,0],"acting":[4,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.e","version":"74'56","reported_seq":322,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.390297+0000","last_change":"2024-04-26T01:08:42.066150+0000","last_active":"2024-04-26T01:08:43.390297+0000","last_peered":"2024-04-26T01:08:43.390297+0000","last_clean":"2024-04-26T01:08:43.390297+0000","last_became_active":"2024-04-26T01:08:11.054010+0000","last_became_peered":"2024-04-26T01:08:11.054010+0000","last_unstale":"2024-04-26T01:08:43.390297+0000","last_undegraded":"2024-04-26T01:08:43.390297+0000","last_fullsized":"2024-04-26T01:08:43.390297+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:25.162677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021227889999999999,"stat_sum":{"num_bytes":5111808,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":16042,"num_write":27,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3112960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,7,8,6,12],"acting":[2,9,1,7,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.d","version":"72'29","reported_seq":275,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.290655+0000","last_change":"2024-04-26T01:08:42.052060+0000","last_active":"2024-04-26T01:08:44.290655+0000","last_peered":"2024-04-26T01:08:44.290655+0000","last_clean":"2024-04-26T01:08:44.290655+0000","last_became_active":"2024-04-26T01:07:57.589029+0000","last_became_peered":"2024-04-26T01:07:57.589029+0000","last_unstale":"2024-04-26T01:08:44.290655+0000","last_undegraded":"2024-04-26T01:08:44.290655+0000","last_fullsized":"2024-04-26T01:08:44.290655+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:09.119203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000565436,"stat_sum":{"num_bytes":1523712,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":775,"num_write":15,"num_write_kb":1648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.c","version":"77'92","reported_seq":267,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.419381+0000","last_change":"2024-04-26T01:08:42.110191+0000","last_active":"2024-04-26T01:08:43.419381+0000","last_peered":"2024-04-26T01:08:43.419381+0000","last_clean":"2024-04-26T01:08:43.419381+0000","last_became_active":"2024-04-26T01:08:11.076105+0000","last_became_peered":"2024-04-26T01:08:11.076105+0000","last_unstale":"2024-04-26T01:08:43.419381+0000","last_undegraded":"2024-04-26T01:08:43.419381+0000","last_fullsized":"2024-04-26T01:08:43.419381+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:02.907098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013337010000000001,"stat_sum":{"num_bytes":15106048,"num_objects":7,"num_object_clones":2,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":45,"num_read_kb":23107,"num_write":86,"num_write_kb":11888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4849664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0,10,6,12,5],"acting":[9,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.b","version":"71'63","reported_seq":312,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.275532+0000","last_change":"2024-04-26T01:08:42.059965+0000","last_active":"2024-04-26T01:08:43.275532+0000","last_peered":"2024-04-26T01:08:43.275532+0000","last_clean":"2024-04-26T01:08:43.275532+0000","last_became_active":"2024-04-26T01:07:57.596133+0000","last_became_peered":"2024-04-26T01:07:57.596133+0000","last_unstale":"2024-04-26T01:08:43.275532+0000","last_undegraded":"2024-04-26T01:08:43.275532+0000","last_fullsized":"2024-04-26T01:08:43.275532+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:25.237963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029999380000000002,"stat_sum":{"num_bytes":8192000,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":18874,"num_write":33,"num_write_kb":11424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.a","version":"74'14","reported_seq":267,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.861040+0000","last_change":"2024-04-26T01:08:42.290581+0000","last_active":"2024-04-26T01:08:44.861040+0000","last_peered":"2024-04-26T01:08:44.861040+0000","last_clean":"2024-04-26T01:08:44.861040+0000","last_became_active":"2024-04-26T01:08:11.250401+0000","last_became_peered":"2024-04-26T01:08:11.250401+0000","last_unstale":"2024-04-26T01:08:44.861040+0000","last_undegraded":"2024-04-26T01:08:44.861040+0000","last_fullsized":"2024-04-26T01:08:44.861040+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:11.820230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001080372,"stat_sum":{"num_bytes":1490944,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":4371,"num_write":7,"num_write_kb":1456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,10],"acting":[3,7,1,9,14,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.9","version":"0'0","reported_seq":234,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.284994+0000","last_change":"2024-04-26T01:08:42.059468+0000","last_active":"2024-04-26T01:08:43.284994+0000","last_peered":"2024-04-26T01:08:43.284994+0000","last_clean":"2024-04-26T01:08:43.284994+0000","last_became_active":"2024-04-26T01:07:57.584830+0000","last_became_peered":"2024-04-26T01:07:57.584830+0000","last_unstale":"2024-04-26T01:08:43.284994+0000","last_undegraded":"2024-04-26T01:08:43.284994+0000","last_fullsized":"2024-04-26T01:08:43.284994+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:11.015264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010981879999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.8","version":"74'44","reported_seq":301,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.317446+0000","last_change":"2024-04-26T01:08:42.109999+0000","last_active":"2024-04-26T01:08:44.317446+0000","last_peered":"2024-04-26T01:08:44.317446+0000","last_clean":"2024-04-26T01:08:44.317446+0000","last_became_active":"2024-04-26T01:07:57.586870+0000","last_became_peered":"2024-04-26T01:07:57.586870+0000","last_unstale":"2024-04-26T01:08:44.317446+0000","last_undegraded":"2024-04-26T01:08:44.317446+0000","last_fullsized":"2024-04-26T01:08:44.317446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:21.490320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023345589999999999,"stat_sum":{"num_bytes":4816896,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":30,"num_read_kb":9202,"num_write":26,"num_write_kb":10400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"1.0","version":"27'157","reported_seq":313,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.859975+0000","last_change":"2024-04-26T01:07:46.409257+0000","last_active":"2024-04-26T01:08:44.859975+0000","last_peered":"2024-04-26T01:08:44.859975+0000","last_clean":"2024-04-26T01:08:44.859975+0000","last_became_active":"2024-04-26T01:07:46.407301+0000","last_became_peered":"2024-04-26T01:07:46.407301+0000","last_unstale":"2024-04-26T01:08:44.859975+0000","last_undegraded":"2024-04-26T01:08:44.859975+0000","last_fullsized":"2024-04-26T01:08:44.859975+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_clean_scrub_stamp":"2024-04-26T01:07:45.031420+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.371437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":114,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.317355+0000","last_change":"2024-04-26T01:07:48.145129+0000","last_active":"2024-04-26T01:08:44.317355+0000","last_peered":"2024-04-26T01:08:44.317355+0000","last_clean":"2024-04-26T01:08:44.317355+0000","last_became_active":"2024-04-26T01:07:48.144313+0000","last_became_peered":"2024-04-26T01:07:48.144313+0000","last_unstale":"2024-04-26T01:08:44.317355+0000","last_undegraded":"2024-04-26T01:08:44.317355+0000","last_fullsized":"2024-04-26T01:08:44.317355+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:03.023694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"73'37","reported_seq":307,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.860201+0000","last_change":"2024-04-26T01:08:42.065755+0000","last_active":"2024-04-26T01:08:44.860201+0000","last_peered":"2024-04-26T01:08:44.860201+0000","last_clean":"2024-04-26T01:08:44.860201+0000","last_became_active":"2024-04-26T01:08:11.059109+0000","last_became_peered":"2024-04-26T01:08:11.059109+0000","last_unstale":"2024-04-26T01:08:44.860201+0000","last_undegraded":"2024-04-26T01:08:44.860201+0000","last_fullsized":"2024-04-26T01:08:44.860201+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:19.067602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015211230000000001,"stat_sum":{"num_bytes":4866048,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":1832,"num_write":44,"num_write_kb":6208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,14],"acting":[13,2,3,6,15,1,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":113,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.860691+0000","last_change":"2024-04-26T01:07:48.149275+0000","last_active":"2024-04-26T01:08:44.860691+0000","last_peered":"2024-04-26T01:08:44.860691+0000","last_clean":"2024-04-26T01:08:44.860691+0000","last_became_active":"2024-04-26T01:07:48.148395+0000","last_became_peered":"2024-04-26T01:07:48.148395+0000","last_unstale":"2024-04-26T01:08:44.860691+0000","last_undegraded":"2024-04-26T01:08:44.860691+0000","last_fullsized":"2024-04-26T01:08:44.860691+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:49.139938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"76'95","reported_seq":373,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.275391+0000","last_change":"2024-04-26T01:08:42.061538+0000","last_active":"2024-04-26T01:08:43.275391+0000","last_peered":"2024-04-26T01:08:43.275391+0000","last_clean":"2024-04-26T01:08:43.275391+0000","last_became_active":"2024-04-26T01:08:11.061821+0000","last_became_peered":"2024-04-26T01:08:11.061821+0000","last_unstale":"2024-04-26T01:08:43.275391+0000","last_undegraded":"2024-04-26T01:08:43.275391+0000","last_fullsized":"2024-04-26T01:08:43.275391+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:04.412956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048083900000000001,"stat_sum":{"num_bytes":9191424,"num_objects":6,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":34,"num_read_kb":8847,"num_write":92,"num_write_kb":14720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3522560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,9,5,10,2,13],"acting":[4,7,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":113,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.859803+0000","last_change":"2024-04-26T01:07:48.146870+0000","last_active":"2024-04-26T01:08:44.859803+0000","last_peered":"2024-04-26T01:08:44.859803+0000","last_clean":"2024-04-26T01:08:44.859803+0000","last_became_active":"2024-04-26T01:07:48.145991+0000","last_became_peered":"2024-04-26T01:07:48.145991+0000","last_unstale":"2024-04-26T01:08:44.859803+0000","last_undegraded":"2024-04-26T01:08:44.859803+0000","last_fullsized":"2024-04-26T01:08:44.859803+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:13.523679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"74'77","reported_seq":324,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.285630+0000","last_change":"2024-04-26T01:08:42.059479+0000","last_active":"2024-04-26T01:08:43.285630+0000","last_peered":"2024-04-26T01:08:43.285630+0000","last_clean":"2024-04-26T01:08:43.285630+0000","last_became_active":"2024-04-26T01:07:57.587743+0000","last_became_peered":"2024-04-26T01:07:57.587743+0000","last_unstale":"2024-04-26T01:08:43.285630+0000","last_undegraded":"2024-04-26T01:08:43.285630+0000","last_fullsized":"2024-04-26T01:08:43.285630+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:06.308832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001306572,"stat_sum":{"num_bytes":11862016,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":3324,"num_write":62,"num_write_kb":20144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":113,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.860989+0000","last_change":"2024-04-26T01:07:48.147756+0000","last_active":"2024-04-26T01:08:44.860989+0000","last_peered":"2024-04-26T01:08:44.860989+0000","last_clean":"2024-04-26T01:08:44.860989+0000","last_became_active":"2024-04-26T01:07:48.147003+0000","last_became_peered":"2024-04-26T01:07:48.147003+0000","last_unstale":"2024-04-26T01:08:44.860989+0000","last_undegraded":"2024-04-26T01:08:44.860989+0000","last_fullsized":"2024-04-26T01:08:44.860989+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:05.491411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"72'20","reported_seq":264,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-26T01:08:44.859987+0000","last_change":"2024-04-26T01:08:42.065635+0000","last_active":"2024-04-26T01:08:44.859987+0000","last_peered":"2024-04-26T01:08:44.859987+0000","last_clean":"2024-04-26T01:08:44.859987+0000","last_became_active":"2024-04-26T01:07:57.310765+0000","last_became_peered":"2024-04-26T01:07:57.310765+0000","last_unstale":"2024-04-26T01:08:44.859987+0000","last_undegraded":"2024-04-26T01:08:44.859987+0000","last_fullsized":"2024-04-26T01:08:44.859987+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.890017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001596313,"stat_sum":{"num_bytes":3506176,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":4325,"num_write":35,"num_write_kb":4960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"3.5","version":"74'50","reported_seq":291,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.300439+0000","last_change":"2024-04-26T01:08:42.101123+0000","last_active":"2024-04-26T01:08:43.300439+0000","last_peered":"2024-04-26T01:08:43.300439+0000","last_clean":"2024-04-26T01:08:43.300439+0000","last_became_active":"2024-04-26T01:07:57.588984+0000","last_became_peered":"2024-04-26T01:07:57.588984+0000","last_unstale":"2024-04-26T01:08:43.300439+0000","last_undegraded":"2024-04-26T01:08:43.300439+0000","last_fullsized":"2024-04-26T01:08:43.300439+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:02.184107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083330600000000002,"stat_sum":{"num_bytes":4587520,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":20,"num_read_kb":17724,"num_write":25,"num_write_kb":10784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":111,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.275406+0000","last_change":"2024-04-26T01:07:48.465954+0000","last_active":"2024-04-26T01:08:43.275406+0000","last_peered":"2024-04-26T01:08:43.275406+0000","last_clean":"2024-04-26T01:08:43.275406+0000","last_became_active":"2024-04-26T01:07:48.465232+0000","last_became_peered":"2024-04-26T01:07:48.465232+0000","last_unstale":"2024-04-26T01:08:43.275406+0000","last_undegraded":"2024-04-26T01:08:43.275406+0000","last_fullsized":"2024-04-26T01:08:43.275406+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:49.270720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"71'66","reported_seq":309,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.286580+0000","last_change":"2024-04-26T01:08:42.062747+0000","last_active":"2024-04-26T01:08:43.286580+0000","last_peered":"2024-04-26T01:08:43.286580+0000","last_clean":"2024-04-26T01:08:43.286580+0000","last_became_active":"2024-04-26T01:07:57.578181+0000","last_became_peered":"2024-04-26T01:07:57.578181+0000","last_unstale":"2024-04-26T01:08:43.286580+0000","last_undegraded":"2024-04-26T01:08:43.286580+0000","last_fullsized":"2024-04-26T01:08:43.286580+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:13.423235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068971699999999996,"stat_sum":{"num_bytes":6389760,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":6899,"num_write":70,"num_write_kb":6752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":112,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.284534+0000","last_change":"2024-04-26T01:07:48.145645+0000","last_active":"2024-04-26T01:08:43.284534+0000","last_peered":"2024-04-26T01:08:43.284534+0000","last_clean":"2024-04-26T01:08:43.284534+0000","last_became_active":"2024-04-26T01:07:48.145376+0000","last_became_peered":"2024-04-26T01:07:48.145376+0000","last_unstale":"2024-04-26T01:08:43.284534+0000","last_undegraded":"2024-04-26T01:08:43.284534+0000","last_fullsized":"2024-04-26T01:08:43.284534+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.310599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"77'87","reported_seq":385,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.289248+0000","last_change":"2024-04-26T01:08:42.061999+0000","last_active":"2024-04-26T01:08:43.289248+0000","last_peered":"2024-04-26T01:08:43.289248+0000","last_clean":"2024-04-26T01:08:43.289248+0000","last_became_active":"2024-04-26T01:08:11.064770+0000","last_became_peered":"2024-04-26T01:08:11.064770+0000","last_unstale":"2024-04-26T01:08:43.289248+0000","last_undegraded":"2024-04-26T01:08:43.289248+0000","last_fullsized":"2024-04-26T01:08:43.289248+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:09.006645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049855719999999997,"stat_sum":{"num_bytes":13926400,"num_objects":8,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":76,"num_read_kb":20738,"num_write":80,"num_write_kb":21920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,14],"acting":[4,6,9,0,8,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":111,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.287174+0000","last_change":"2024-04-26T01:07:48.145909+0000","last_active":"2024-04-26T01:08:43.287174+0000","last_peered":"2024-04-26T01:08:43.287174+0000","last_clean":"2024-04-26T01:08:43.287174+0000","last_became_active":"2024-04-26T01:07:48.145512+0000","last_became_peered":"2024-04-26T01:07:48.145512+0000","last_unstale":"2024-04-26T01:08:43.287174+0000","last_undegraded":"2024-04-26T01:08:43.287174+0000","last_fullsized":"2024-04-26T01:08:43.287174+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:16.344494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"77'63","reported_seq":313,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.297905+0000","last_change":"2024-04-26T01:08:42.064126+0000","last_active":"2024-04-26T01:08:43.297905+0000","last_peered":"2024-04-26T01:08:43.297905+0000","last_clean":"2024-04-26T01:08:43.297905+0000","last_became_active":"2024-04-26T01:07:57.597503+0000","last_became_peered":"2024-04-26T01:07:57.597503+0000","last_unstale":"2024-04-26T01:08:43.297905+0000","last_undegraded":"2024-04-26T01:08:43.297905+0000","last_fullsized":"2024-04-26T01:08:43.297905+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:17.591570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060259500000000004,"stat_sum":{"num_bytes":11534336,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":9290,"num_write":59,"num_write_kb":13104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13","length":"2"},{"start":"16","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":110,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T01:08:43.282760+0000","last_change":"2024-04-26T01:07:48.149017+0000","last_active":"2024-04-26T01:08:43.282760+0000","last_peered":"2024-04-26T01:08:43.282760+0000","last_clean":"2024-04-26T01:08:43.282760+0000","last_became_active":"2024-04-26T01:07:48.148023+0000","last_became_peered":"2024-04-26T01:07:48.148023+0000","last_unstale":"2024-04-26T01:08:43.282760+0000","last_undegraded":"2024-04-26T01:08:43.282760+0000","last_fullsized":"2024-04-26T01:08:43.282760+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:07.554062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":116391936,"num_objects":63,"num_object_clones":17,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":4,"num_read":426,"num_read_kb":162938,"num_write":747,"num_write_kb":160416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":17498112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200904704,"data_stored":242757632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1802,"internal_metadata":0},"log_size":897,"ondisk_log_size":897,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":23216,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":21,"seq":90194313232,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38900,"kb_used_data":11880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332940,"statfs":{"total":96636764160,"available":96596930560,"internally_reserved":0,"allocated":12165120,"data_stored":15144447,"data_compressed":168835,"data_compressed_allocated":229376,"data_compressed_original":458752,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,3,4,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280526,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45368,"kb_used_data":14188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326472,"statfs":{"total":96636764160,"available":96590307328,"internally_reserved":0,"allocated":14528512,"data_stored":17596322,"data_compressed":177378,"data_compressed_allocated":266240,"data_compressed_original":962560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280528,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45324,"kb_used_data":14140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326516,"statfs":{"total":96636764160,"available":96590352384,"internally_reserved":0,"allocated":14479360,"data_stored":17323519,"data_compressed":168835,"data_compressed_allocated":229376,"data_compressed_original":458752,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,8,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313230,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54820,"kb_used_data":19480,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94317020,"statfs":{"total":96636764160,"available":96580628480,"internally_reserved":0,"allocated":19947520,"data_stored":23723394,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313231,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46752,"kb_used_data":15572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325088,"statfs":{"total":96636764160,"available":96588890112,"internally_reserved":0,"allocated":15945728,"data_stored":18595746,"data_compressed":177378,"data_compressed_allocated":266240,"data_compressed_original":962560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313232,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42088,"kb_used_data":10908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329752,"statfs":{"total":96636764160,"available":96593666048,"internally_reserved":0,"allocated":11169792,"data_stored":13282690,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313230,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49972,"kb_used_data":18784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321868,"statfs":{"total":96636764160,"available":96585592832,"internally_reserved":0,"allocated":19234816,"data_stored":20725122,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313230,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48128,"kb_used_data":16948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323712,"statfs":{"total":96636764160,"available":96587481088,"internally_reserved":0,"allocated":17354752,"data_stored":20376962,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":21,"seq":90194313230,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41764,"kb_used_data":14736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330076,"statfs":{"total":96636764160,"available":96593997824,"internally_reserved":0,"allocated":15089664,"data_stored":18234754,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313231,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42656,"kb_used_data":15628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329184,"statfs":{"total":96636764160,"available":96593084416,"internally_reserved":0,"allocated":16003072,"data_stored":20123010,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,5,6,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313230,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36444,"kb_used_data":9420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335396,"statfs":{"total":96636764160,"available":96599445504,"internally_reserved":0,"allocated":9646080,"data_stored":11474281,"data_compressed":162730,"data_compressed_allocated":221184,"data_compressed_original":442368,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":0,"seq":257698037765,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313232,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40780,"kb_used_data":9592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331060,"statfs":{"total":96636764160,"available":96595005440,"internally_reserved":0,"allocated":9822208,"data_stored":11269481,"data_compressed":162730,"data_compressed_allocated":221184,"data_compressed_original":442368,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[5,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313231,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42320,"kb_used_data":15292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329520,"statfs":{"total":96636764160,"available":96593428480,"internally_reserved":0,"allocated":15659008,"data_stored":16665986,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280528,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45284,"kb_used_data":14104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326556,"statfs":{"total":96636764160,"available":96590393344,"internally_reserved":0,"allocated":14442496,"data_stored":16731522,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280527,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34808,"kb_used_data":3628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337032,"statfs":{"total":96636764160,"available":96601120768,"internally_reserved":0,"allocated":3715072,"data_stored":4726146,"data_compressed":165770,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,7,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19070976,"data_stored":23093248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13565952,"data_stored":16101376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15007744,"data_stored":17436672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2838528,"data_stored":4096000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18358272,"data_stored":20094976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":13590528,"data_stored":16683008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16478208,"data_stored":19746816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11276288,"data_stored":14503936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14213120,"data_stored":17604608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":15126528,"data_stored":19492864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":9363456,"data_stored":11436032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4374528,"data_stored":4374528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1802,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":8974336,"data_stored":12967936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":13590528,"data_stored":16437248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":14782464,"data_stored":16035840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":10293248,"data_stored":12652544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:08:46.088 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-26T01:08:46.089 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T01:08:46.089 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-04-26T01:08:46.089 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 15, 11] 2024-04-26T01:08:46.089 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-26T01:08:46.089 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:08:46.125 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:46.125 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:46.140 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:08:46.142 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:46.137+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:46.151 INFO:tasks.ceph.osd.3:Stopped 2024-04-26T01:08:46.151 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 3 2024-04-26T01:08:46.243 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:46.241+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:46.304 INFO:tasks.rados.rados.0.smithi164.stdout:341: done (1 left) 2024-04-26T01:08:46.304 INFO:tasks.rados.rados.0.smithi164.stdout:342: done (0 left) 2024-04-26T01:08:46.305 INFO:tasks.rados.rados.0.smithi164.stdout:343: read oid 42 snap -1 2024-04-26T01:08:46.305 INFO:tasks.rados.rados.0.smithi164.stdout:343: expect (ObjNum 111 snap 14 seq_num 111) 2024-04-26T01:08:46.305 INFO:tasks.rados.rados.0.smithi164.stdout:344: read oid 20 snap -1 2024-04-26T01:08:46.305 INFO:tasks.rados.rados.0.smithi164.stdout:344: expect (ObjNum 69 snap 4 seq_num 69) 2024-04-26T01:08:46.305 INFO:tasks.rados.rados.0.smithi164.stdout:345: setattr oid 50 current snap is 26 2024-04-26T01:08:46.306 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:46.306 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:46.315 INFO:tasks.rados.rados.0.smithi164.stdout:346: copy_from oid 37 from oid 27 current snap is 26 2024-04-26T01:08:46.315 INFO:tasks.rados.rados.0.smithi164.stdout:347: read oid 6 snap -1 2024-04-26T01:08:46.316 INFO:tasks.rados.rados.0.smithi164.stdout:347: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-26T01:08:46.316 INFO:tasks.rados.rados.0.smithi164.stdout:348: snap_remove snap 19 2024-04-26T01:08:46.329 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:08:46.344 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:46.340+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:46.446 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:46.441+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:46.446 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:46.446 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:46.461 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:08:46.473 INFO:tasks.ceph.mon.a.smithi008.stderr:2024-04-26T01:08:46.468+0000 7f99831a5640 -1 mon.a@0(leader).osd e80 definitely_dead 0 2024-04-26T01:08:46.473 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3 is already down. 2024-04-26T01:08:46.492 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15] 2024-04-26T01:08:46.492 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 3 2024-04-26T01:08:46.578 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:46.576+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:46.584 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:46.584 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:46.599 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:08:46.647 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:46.647+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:46.719 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:46.719 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:46.733 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:08:46.748 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:46.744+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:46.849 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:46.849+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:46.858 INFO:tasks.ceph.mon.a.smithi008.stderr:2024-04-26T01:08:46.856+0000 7f99831a5640 -1 mon.a@0(leader).osd e80 definitely_dead 0 2024-04-26T01:08:46.899 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:46.899 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:46.913 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:08:46.917 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:08:46.959 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:46.959+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:47.037 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:47.037 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:47.051 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:08:47.052 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:47.051+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:47.079 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:47.079 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:47.108 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:47.108 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:47.137 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:47.138 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:47.138 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-26T01:08:47.151 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:47.148+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:47.166 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:47.166 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:47.197 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:47.197 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:47.219 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:47.220 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:47.229 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:47.230 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:47.233 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:08:47.252 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:47.251+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:47.262 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:47.262 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:47.295 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:47.296 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:47.303 INFO:teuthology.orchestra.run.smithi008.stderr:marked out osd.3. 2024-04-26T01:08:47.310 INFO:tasks.rados.rados.0.smithi164.stdout:348: done (5 left) 2024-04-26T01:08:47.310 INFO:tasks.rados.rados.0.smithi164.stdout:349: read oid 41 snap 22 2024-04-26T01:08:47.310 INFO:tasks.rados.rados.0.smithi164.stdout:349: expect (ObjNum 137 snap 21 seq_num 137) 2024-04-26T01:08:47.314 INFO:tasks.rados.rados.0.smithi164.stdout:347: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-26T01:08:47.317 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 50 v 78 (ObjNum 673 snap 0 seq_num 1564696432) dirty exists 2024-04-26T01:08:47.317 INFO:tasks.rados.rados.0.smithi164.stdout:343: expect (ObjNum 111 snap 14 seq_num 111) 2024-04-26T01:08:47.323 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-26T01:08:47.342 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-04-26T01:08:47.342 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi164.front.sepia.ceph.com 2024-04-26T01:08:47.342 DEBUG:teuthology.orchestra.run.smithi164:> type ceph-objectstore-tool 2024-04-26T01:08:47.347 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-26T01:08:47.348 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-04-26T01:08:47.348 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:47.349 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:47.353 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:47.349+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:47.368 INFO:teuthology.orchestra.run.smithi116.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:47.368 INFO:teuthology.orchestra.run.smithi116.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:47.372 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:47.373 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:47.383 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:08:47.398 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:08:47.398 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:08:47.422 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:47.422 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:47.449 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:47.449 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:47.453 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:47.452+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:47.476 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:47.476 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:47.476 INFO:tasks.rados.rados.0.smithi164.stdout:343: done (5 left) 2024-04-26T01:08:47.477 INFO:tasks.rados.rados.0.smithi164.stdout:345: done (4 left) 2024-04-26T01:08:47.477 INFO:tasks.rados.rados.0.smithi164.stdout:347: done (3 left) 2024-04-26T01:08:47.477 INFO:tasks.rados.rados.0.smithi164.stdout:350: rollback oid 15 current snap is 26 2024-04-26T01:08:47.477 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 15 to 22 2024-04-26T01:08:47.477 INFO:tasks.rados.rados.0.smithi164.stdout:351: setattr oid 26 current snap is 26 2024-04-26T01:08:47.480 INFO:tasks.rados.rados.0.smithi164.stdout:352: rollback oid 44 current snap is 26 2024-04-26T01:08:47.480 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 44 to 25 2024-04-26T01:08:47.480 INFO:tasks.rados.rados.0.smithi164.stdout:353: rollback oid 3 current snap is 26 2024-04-26T01:08:47.480 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 3 to 24 2024-04-26T01:08:47.480 INFO:tasks.rados.rados.0.smithi164.stdout:354: copy_from oid 8 from oid 5 current snap is 26 2024-04-26T01:08:47.480 INFO:tasks.rados.rados.0.smithi164.stdout:355: delete oid 31 current snap is 26 2024-04-26T01:08:47.487 INFO:tasks.rados.rados.0.smithi164.stdout:350: finishing rollback tid 0 to smithi16428812-15 2024-04-26T01:08:47.490 INFO:tasks.rados.rados.0.smithi164.stdout:350: finishing rollback tid 1 to smithi16428812-15 2024-04-26T01:08:47.494 INFO:tasks.rados.rados.0.smithi164.stdout:352: finishing rollback tid 0 to smithi16428812-44 2024-04-26T01:08:47.495 INFO:tasks.rados.rados.0.smithi164.stdout:353: finishing rollback tid 0 to smithi16428812-3 2024-04-26T01:08:47.498 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 26 v 99 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-26T01:08:47.499 INFO:tasks.rados.rados.0.smithi164.stdout:352: finishing rollback tid 1 to smithi16428812-44 2024-04-26T01:08:47.499 INFO:tasks.rados.rados.0.smithi164.stdout:353: finishing rollback tid 1 to smithi16428812-3 2024-04-26T01:08:47.500 INFO:tasks.rados.rados.0.smithi164.stdout:351: done (8 left) 2024-04-26T01:08:47.500 INFO:tasks.rados.rados.0.smithi164.stdout:355: done (7 left) 2024-04-26T01:08:47.500 INFO:tasks.rados.rados.0.smithi164.stdout:356: copy_from oid 45 from oid 9 current snap is 26 2024-04-26T01:08:47.501 INFO:tasks.rados.rados.0.smithi164.stdout:357: delete oid 43 current snap is 26 2024-04-26T01:08:47.503 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:47.500+0000 7f39e2ffd640 0 -- 172.21.15.8:0/3849621092 >> [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] conn(0x7f39e421e390 msgr2=0x7f39e4220780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:08:47.503 INFO:tasks.rados.rados.0.smithi164.stdout:353: finishing rollback tid 2 to smithi16428812-3 2024-04-26T01:08:47.503 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 3 v 52 (ObjNum 127 snap 19 seq_num 127) dirty exists 2024-04-26T01:08:47.507 INFO:tasks.rados.rados.0.smithi164.stdout:352: finishing rollback tid 2 to smithi16428812-44 2024-04-26T01:08:47.507 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 44 v 71 (ObjNum 113 snap 14 seq_num 113) dirty exists 2024-04-26T01:08:47.515 INFO:tasks.rados.rados.0.smithi164.stdout:349: expect (ObjNum 137 snap 21 seq_num 137) 2024-04-26T01:08:47.516 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:47.516 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:08:47.516 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:08:47.516 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:08:47.516 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.516 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:105 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:47.516 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:08:47.516 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 48.090061529000003, 2024-04-26T01:08:47.516 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017139178000000001, 2024-04-26T01:08:47.516 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.516 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.516 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:47.517 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.517 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.517 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.517 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 105 2024-04-26T01:08:47.517 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.517 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.517 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.517 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.517 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:08:47.517 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.517 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.517 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.517 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.517 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:08:47.517 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.518 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.518 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.518 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.518 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425522+0000", 2024-04-26T01:08:47.518 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5430000000000001e-06 2024-04-26T01:08:47.518 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.518 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.518 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.518 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425529+0000", 2024-04-26T01:08:47.518 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2570000000000001e-06 2024-04-26T01:08:47.518 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.518 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.519 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.519 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425548+0000", 2024-04-26T01:08:47.519 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8674000000000001e-05 2024-04-26T01:08:47.519 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.519 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.519 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.519 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425560+0000", 2024-04-26T01:08:47.519 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2683e-05 2024-04-26T01:08:47.519 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.519 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.519 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.519 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428152+0000", 2024-04-26T01:08:47.519 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0025912800000000001 2024-04-26T01:08:47.519 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.520 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.520 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:47.520 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428234+0000", 2024-04-26T01:08:47.520 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2105000000000001e-05 2024-04-26T01:08:47.520 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.520 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.520 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.520 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438095+0000", 2024-04-26T01:08:47.520 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0098614180000000003 2024-04-26T01:08:47.520 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.520 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.520 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.520 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438205+0000", 2024-04-26T01:08:47.520 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000109849 2024-04-26T01:08:47.521 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.521 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.521 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.521 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442659+0000", 2024-04-26T01:08:47.521 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0044543689999999997 2024-04-26T01:08:47.521 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.521 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.521 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.521 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.521 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.521 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:117 3.0s0 3:00e538ad:::smithi16428812-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:47.521 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:08:47.521 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 48.076958533000003, 2024-04-26T01:08:47.521 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017480893000000001, 2024-04-26T01:08:47.522 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.522 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.522 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:47.522 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.522 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.522 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.522 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 117 2024-04-26T01:08:47.522 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.522 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.522 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.522 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.522 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:08:47.522 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.522 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.523 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.523 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.523 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:08:47.523 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.523 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.523 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.523 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438624+0000", 2024-04-26T01:08:47.523 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.6000000000000003e-07 2024-04-26T01:08:47.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.523 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.523 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.523 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438628+0000", 2024-04-26T01:08:47.523 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8029999999999998e-06 2024-04-26T01:08:47.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.524 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.524 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.524 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438635+0000", 2024-04-26T01:08:47.524 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2799999999999998e-06 2024-04-26T01:08:47.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.524 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.524 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.524 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438641+0000", 2024-04-26T01:08:47.524 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5459999999999997e-06 2024-04-26T01:08:47.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.524 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.524 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.524 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442203+0000", 2024-04-26T01:08:47.524 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003561594 2024-04-26T01:08:47.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.525 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.525 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:47.525 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442287+0000", 2024-04-26T01:08:47.525 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.4443000000000004e-05 2024-04-26T01:08:47.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.525 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.525 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.525 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451430+0000", 2024-04-26T01:08:47.525 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0091427130000000002 2024-04-26T01:08:47.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.525 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.525 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.525 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451500+0000", 2024-04-26T01:08:47.525 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9342000000000001e-05 2024-04-26T01:08:47.526 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.526 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.526 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.526 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.456104+0000", 2024-04-26T01:08:47.526 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0046045119999999998 2024-04-26T01:08:47.526 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.526 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.526 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.526 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.526 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.526 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:152 3.0s0 3:0230bbd3:::smithi16428812-50:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:47.526 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.489313+0000", 2024-04-26T01:08:47.526 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 48.026269036000002, 2024-04-26T01:08:47.526 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.015629423, 2024-04-26T01:08:47.527 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.527 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.527 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:47.527 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.527 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.527 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.527 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 152 2024-04-26T01:08:47.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.527 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.527 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.527 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.528 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489313+0000", 2024-04-26T01:08:47.528 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.528 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.528 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.528 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.528 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489313+0000", 2024-04-26T01:08:47.528 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.528 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.528 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.528 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.528 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489314+0000", 2024-04-26T01:08:47.528 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2399999999999996e-07 2024-04-26T01:08:47.528 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.529 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.529 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489318+0000", 2024-04-26T01:08:47.529 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3800000000000004e-06 2024-04-26T01:08:47.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.529 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.529 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489326+0000", 2024-04-26T01:08:47.529 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5459999999999992e-06 2024-04-26T01:08:47.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.529 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.529 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489335+0000", 2024-04-26T01:08:47.529 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9050000000000003e-06 2024-04-26T01:08:47.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.530 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.530 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.491271+0000", 2024-04-26T01:08:47.530 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0019354579999999999 2024-04-26T01:08:47.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.530 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.530 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.491416+0000", 2024-04-26T01:08:47.530 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000145396 2024-04-26T01:08:47.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.530 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:47.530 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492139+0000", 2024-04-26T01:08:47.531 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00072277400000000003 2024-04-26T01:08:47.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.531 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:47.531 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492918+0000", 2024-04-26T01:08:47.531 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00077903499999999997 2024-04-26T01:08:47.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.531 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:47.531 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504860+0000", 2024-04-26T01:08:47.531 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011941712 2024-04-26T01:08:47.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.531 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.532 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504942+0000", 2024-04-26T01:08:47.532 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2293000000000006e-05 2024-04-26T01:08:47.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.532 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.532 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:153 3.0s0 3:0230bbd3:::smithi16428812-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:47.532 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:08:47.532 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 48.026235370999999, 2024-04-26T01:08:47.532 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017766607, 2024-04-26T01:08:47.532 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.532 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.532 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:47.532 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.533 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.533 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.533 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 153 2024-04-26T01:08:47.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.533 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.533 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.533 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:08:47.533 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.533 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.533 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:08:47.533 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.534 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.534 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.534 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489347+0000", 2024-04-26T01:08:47.534 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9999999999999997e-07 2024-04-26T01:08:47.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.534 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.534 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.534 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489351+0000", 2024-04-26T01:08:47.534 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7309999999999999e-06 2024-04-26T01:08:47.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.534 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.534 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.535 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489359+0000", 2024-04-26T01:08:47.535 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.6619999999999994e-06 2024-04-26T01:08:47.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.535 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.535 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.535 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489366+0000", 2024-04-26T01:08:47.535 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.798e-06 2024-04-26T01:08:47.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.535 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.535 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.535 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492454+0000", 2024-04-26T01:08:47.535 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0030882399999999999 2024-04-26T01:08:47.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.535 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.536 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:47.536 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492562+0000", 2024-04-26T01:08:47.536 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107948 2024-04-26T01:08:47.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.536 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.536 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.536 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504997+0000", 2024-04-26T01:08:47.536 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012434176 2024-04-26T01:08:47.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.536 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.536 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.536 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.505136+0000", 2024-04-26T01:08:47.536 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013994300000000001 2024-04-26T01:08:47.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.537 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.537 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.537 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.507113+0000", 2024-04-26T01:08:47.537 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0019764090000000002 2024-04-26T01:08:47.537 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.537 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.537 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.537 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.537 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:284 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~753664] snapc 4=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:47.537 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:04.530659+0000", 2024-04-26T01:08:47.537 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 42.984922222999998, 2024-04-26T01:08:47.537 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.015592622, 2024-04-26T01:08:47.537 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.538 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.538 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:47.538 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.538 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.538 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.538 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 284 2024-04-26T01:08:47.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.538 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.538 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.538 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.530659+0000", 2024-04-26T01:08:47.538 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.539 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.539 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.530659+0000", 2024-04-26T01:08:47.539 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.539 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.539 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.539 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.530663+0000", 2024-04-26T01:08:47.539 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7529999999999999e-06 2024-04-26T01:08:47.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.539 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.539 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.539 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.531697+0000", 2024-04-26T01:08:47.539 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0010340410000000001 2024-04-26T01:08:47.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.540 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.540 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.540 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.531730+0000", 2024-04-26T01:08:47.540 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.3194000000000001e-05 2024-04-26T01:08:47.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.540 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.540 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.540 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.531756+0000", 2024-04-26T01:08:47.540 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5559999999999999e-05 2024-04-26T01:08:47.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.540 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.540 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.540 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.531934+0000", 2024-04-26T01:08:47.540 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017789499999999999 2024-04-26T01:08:47.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.541 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.541 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.532110+0000", 2024-04-26T01:08:47.541 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017660500000000001 2024-04-26T01:08:47.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.541 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:47.541 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.534200+0000", 2024-04-26T01:08:47.541 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0020893299999999999 2024-04-26T01:08:47.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.541 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:47.541 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.541721+0000", 2024-04-26T01:08:47.542 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0075214430000000001 2024-04-26T01:08:47.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.542 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:47.542 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.546130+0000", 2024-04-26T01:08:47.542 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0044085820000000003 2024-04-26T01:08:47.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.542 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.542 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.546252+0000", 2024-04-26T01:08:47.542 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012221899999999999 2024-04-26T01:08:47.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.542 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.543 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.543 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:287 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:47.543 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:04.533969+0000", 2024-04-26T01:08:47.543 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 42.981612300000002, 2024-04-26T01:08:47.543 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.018210147999999999, 2024-04-26T01:08:47.543 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.543 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.543 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:47.543 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.543 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.543 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.543 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 287 2024-04-26T01:08:47.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.544 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.544 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.544 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.544 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533969+0000", 2024-04-26T01:08:47.544 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.544 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.544 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.544 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533969+0000", 2024-04-26T01:08:47.544 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.545 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.545 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.545 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533971+0000", 2024-04-26T01:08:47.545 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3629999999999999e-06 2024-04-26T01:08:47.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.545 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.545 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.545 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533977+0000", 2024-04-26T01:08:47.545 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5799999999999997e-06 2024-04-26T01:08:47.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.545 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.545 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.545 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.534001+0000", 2024-04-26T01:08:47.545 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4006999999999999e-05 2024-04-26T01:08:47.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.546 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.546 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.546 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.534016+0000", 2024-04-26T01:08:47.546 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5160000000000001e-05 2024-04-26T01:08:47.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.546 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.546 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.546 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.545071+0000", 2024-04-26T01:08:47.546 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011054938 2024-04-26T01:08:47.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.546 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.546 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:47.546 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.545349+0000", 2024-04-26T01:08:47.547 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00027734100000000001 2024-04-26T01:08:47.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.547 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.547 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.547 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.549341+0000", 2024-04-26T01:08:47.547 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0039925150000000003 2024-04-26T01:08:47.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.547 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.547 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.547 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.549639+0000", 2024-04-26T01:08:47.547 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00029756800000000001 2024-04-26T01:08:47.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.548 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.548 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.548 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.552180+0000", 2024-04-26T01:08:47.548 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.002540676 2024-04-26T01:08:47.548 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.548 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.548 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.548 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.548 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:371 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:47.548 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.438556+0000", 2024-04-26T01:08:47.548 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.077025728999999, 2024-04-26T01:08:47.548 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.015156252, 2024-04-26T01:08:47.549 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.549 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.549 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:47.549 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.549 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.549 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.549 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 371 2024-04-26T01:08:47.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.549 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.549 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.549 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.549 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438556+0000", 2024-04-26T01:08:47.549 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.550 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.550 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.550 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438556+0000", 2024-04-26T01:08:47.550 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.550 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.550 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.550 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.550 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438557+0000", 2024-04-26T01:08:47.550 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4699999999999999e-06 2024-04-26T01:08:47.550 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.550 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.550 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.550 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438563+0000", 2024-04-26T01:08:47.551 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7699999999999998e-06 2024-04-26T01:08:47.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.551 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.551 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.551 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438608+0000", 2024-04-26T01:08:47.551 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.4898999999999998e-05 2024-04-26T01:08:47.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.551 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.551 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.551 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.438623+0000", 2024-04-26T01:08:47.551 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.47e-05 2024-04-26T01:08:47.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.551 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.551 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.551 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.450782+0000", 2024-04-26T01:08:47.552 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012158983 2024-04-26T01:08:47.552 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.552 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.552 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.552 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.450982+0000", 2024-04-26T01:08:47.552 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000200602 2024-04-26T01:08:47.552 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.552 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.552 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.552 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.453712+0000", 2024-04-26T01:08:47.552 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0027298280000000001 2024-04-26T01:08:47.552 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.552 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.552 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.553 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.553 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:542 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~606208] snapc e=[e,d,c,b,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-04-26T01:08:47.553 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:20.149139+0000", 2024-04-26T01:08:47.553 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 27.366442725999999, 2024-04-26T01:08:47.553 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016978235000000001, 2024-04-26T01:08:47.553 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.553 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.553 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:47.553 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.553 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.553 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.553 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 542 2024-04-26T01:08:47.554 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.554 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.554 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.554 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.554 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.149139+0000", 2024-04-26T01:08:47.554 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.554 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.554 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.554 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.554 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.149139+0000", 2024-04-26T01:08:47.554 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.554 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.554 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.554 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.555 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.149142+0000", 2024-04-26T01:08:47.555 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9639999999999999e-06 2024-04-26T01:08:47.555 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.555 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.555 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.555 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150216+0000", 2024-04-26T01:08:47.555 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001074478 2024-04-26T01:08:47.555 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.555 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.555 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.555 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150251+0000", 2024-04-26T01:08:47.555 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4563000000000001e-05 2024-04-26T01:08:47.555 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.555 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.556 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.556 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150269+0000", 2024-04-26T01:08:47.556 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.806e-05 2024-04-26T01:08:47.556 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.556 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.556 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.556 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150398+0000", 2024-04-26T01:08:47.556 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012864500000000001 2024-04-26T01:08:47.556 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.556 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.556 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "op must wait for map", 2024-04-26T01:08:47.556 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150410+0000", 2024-04-26T01:08:47.556 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2427e-05 2024-04-26T01:08:47.556 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.557 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.557 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.557 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.154650+0000", 2024-04-26T01:08:47.557 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0042403129999999999 2024-04-26T01:08:47.557 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.557 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.557 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.557 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.154806+0000", 2024-04-26T01:08:47.557 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015539199999999999 2024-04-26T01:08:47.557 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.557 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.557 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:47.557 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.157843+0000", 2024-04-26T01:08:47.557 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0030374519999999999 2024-04-26T01:08:47.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.558 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:47.558 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.165692+0000", 2024-04-26T01:08:47.558 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0078487009999999996 2024-04-26T01:08:47.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.558 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:47.558 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.165984+0000", 2024-04-26T01:08:47.558 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00029256099999999999 2024-04-26T01:08:47.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.558 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.559 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.166117+0000", 2024-04-26T01:08:47.559 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013267899999999999 2024-04-26T01:08:47.559 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.559 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.559 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.559 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.559 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:543 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~688128] snapc e=[e,d,c,b,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-04-26T01:08:47.559 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:20.158153+0000", 2024-04-26T01:08:47.559 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 27.357429061000001, 2024-04-26T01:08:47.559 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014883451000000001, 2024-04-26T01:08:47.559 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.559 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.559 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:47.560 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.560 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.560 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.560 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 543 2024-04-26T01:08:47.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.560 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.560 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.560 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.560 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.158153+0000", 2024-04-26T01:08:47.560 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.560 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.560 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.561 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.158153+0000", 2024-04-26T01:08:47.561 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.561 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.561 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.561 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.158156+0000", 2024-04-26T01:08:47.561 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.2969999999999999e-06 2024-04-26T01:08:47.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.561 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.561 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.561 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.159404+0000", 2024-04-26T01:08:47.561 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001248426 2024-04-26T01:08:47.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.562 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.562 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.562 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.159440+0000", 2024-04-26T01:08:47.562 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5284000000000003e-05 2024-04-26T01:08:47.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.562 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.562 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.562 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.159462+0000", 2024-04-26T01:08:47.562 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2682999999999999e-05 2024-04-26T01:08:47.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.562 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.562 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.562 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.160794+0000", 2024-04-26T01:08:47.562 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001331336 2024-04-26T01:08:47.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.563 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.563 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.161097+0000", 2024-04-26T01:08:47.563 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00030376699999999998 2024-04-26T01:08:47.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.563 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:47.563 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.163627+0000", 2024-04-26T01:08:47.563 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0025294900000000001 2024-04-26T01:08:47.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.563 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:47.563 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.172181+0000", 2024-04-26T01:08:47.564 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0085543579999999998 2024-04-26T01:08:47.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.564 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.564 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:47.564 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.172841+0000", 2024-04-26T01:08:47.564 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00065971499999999995 2024-04-26T01:08:47.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.564 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.564 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.564 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.173036+0000", 2024-04-26T01:08:47.564 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000195095 2024-04-26T01:08:47.564 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.564 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.565 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.565 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.565 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.565 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.4s5 55/30 ECSubWrite(tid=39, reqid=osd.8.0:39, at_version=55'26, trim_to=0'0, roll_forward_to=52'24))", 2024-04-26T01:08:47.565 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:21.091694+0000", 2024-04-26T01:08:47.565 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 26.423887443000002, 2024-04-26T01:08:47.565 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.49007189200000001, 2024-04-26T01:08:47.565 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.565 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.565 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:47.565 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.565 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.565 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.565 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091694+0000", 2024-04-26T01:08:47.566 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.566 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.566 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.566 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091694+0000", 2024-04-26T01:08:47.566 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.566 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.566 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.566 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091695+0000", 2024-04-26T01:08:47.566 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.0699999999999996e-07 2024-04-26T01:08:47.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.566 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091753+0000", 2024-04-26T01:08:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7516999999999998e-05 2024-04-26T01:08:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580278+0000", 2024-04-26T01:08:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.48852570099999998 2024-04-26T01:08:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580292+0000", 2024-04-26T01:08:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3671e-05 2024-04-26T01:08:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.568 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.568 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580396+0000", 2024-04-26T01:08:47.568 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010389400000000001 2024-04-26T01:08:47.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.568 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:47.568 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580413+0000", 2024-04-26T01:08:47.568 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7121000000000001e-05 2024-04-26T01:08:47.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.568 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:47.568 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.581731+0000", 2024-04-26T01:08:47.568 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001318294 2024-04-26T01:08:47.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.569 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.569 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.581766+0000", 2024-04-26T01:08:47.569 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4786999999999997e-05 2024-04-26T01:08:47.569 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.569 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.569 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.569 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 58/30 ECSubWriteReply(tid=65, last_complete=58'49, committed=1, applied=1))", 2024-04-26T01:08:47.569 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:24.344800+0000", 2024-04-26T01:08:47.569 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 23.170781735999999, 2024-04-26T01:08:47.569 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.33836068000000002, 2024-04-26T01:08:47.569 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.570 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.570 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:47.570 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.570 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.570 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344800+0000", 2024-04-26T01:08:47.570 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.570 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.570 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344800+0000", 2024-04-26T01:08:47.570 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.571 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.571 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344802+0000", 2024-04-26T01:08:47.571 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.593e-06 2024-04-26T01:08:47.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.571 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.571 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344812+0000", 2024-04-26T01:08:47.571 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0943e-05 2024-04-26T01:08:47.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.571 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.571 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.682937+0000", 2024-04-26T01:08:47.571 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.33812443599999997 2024-04-26T01:08:47.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.572 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.572 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.682952+0000", 2024-04-26T01:08:47.572 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5116000000000001e-05 2024-04-26T01:08:47.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.572 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.572 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.683088+0000", 2024-04-26T01:08:47.572 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000135969 2024-04-26T01:08:47.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.572 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.572 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.683161+0000", 2024-04-26T01:08:47.573 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2622999999999999e-05 2024-04-26T01:08:47.573 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.573 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.573 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.573 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.4s5 60/30 ECSubWrite(tid=58, reqid=osd.8.0:58, at_version=60'44, trim_to=0'0, roll_forward_to=59'40))", 2024-04-26T01:08:47.573 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.181832+0000", 2024-04-26T01:08:47.573 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 21.333749516000001, 2024-04-26T01:08:47.573 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.020381277999999999, 2024-04-26T01:08:47.573 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.573 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.573 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:47.573 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.574 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.574 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181832+0000", 2024-04-26T01:08:47.574 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.574 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.574 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181832+0000", 2024-04-26T01:08:47.574 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.574 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.574 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181833+0000", 2024-04-26T01:08:47.574 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.0999999999999998e-07 2024-04-26T01:08:47.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.575 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.575 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181871+0000", 2024-04-26T01:08:47.575 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8352999999999997e-05 2024-04-26T01:08:47.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.575 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.575 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.182274+0000", 2024-04-26T01:08:47.575 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00040273700000000002 2024-04-26T01:08:47.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.575 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.575 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.182291+0000", 2024-04-26T01:08:47.576 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6949999999999999e-05 2024-04-26T01:08:47.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.576 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.576 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187997+0000", 2024-04-26T01:08:47.576 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.005706198 2024-04-26T01:08:47.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.576 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:47.576 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.188021+0000", 2024-04-26T01:08:47.576 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3652999999999999e-05 2024-04-26T01:08:47.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.576 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:47.577 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.202131+0000", 2024-04-26T01:08:47.577 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014110531000000001 2024-04-26T01:08:47.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.577 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.577 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.202213+0000", 2024-04-26T01:08:47.577 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2145999999999996e-05 2024-04-26T01:08:47.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.577 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.577 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs2 60/44 ECSubWrite(tid=353, reqid=osd.4.0:353, at_version=60'83, trim_to=0'0, roll_forward_to=59'79))", 2024-04-26T01:08:47.577 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.186301+0000", 2024-04-26T01:08:47.578 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 21.329280574999999, 2024-04-26T01:08:47.578 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016791533000000001, 2024-04-26T01:08:47.578 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.578 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.578 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:47.578 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.578 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.578 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186301+0000", 2024-04-26T01:08:47.578 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.578 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.578 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186301+0000", 2024-04-26T01:08:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186304+0000", 2024-04-26T01:08:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4099999999999998e-06 2024-04-26T01:08:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186355+0000", 2024-04-26T01:08:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.1236000000000002e-05 2024-04-26T01:08:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186442+0000", 2024-04-26T01:08:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7143000000000002e-05 2024-04-26T01:08:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186463+0000", 2024-04-26T01:08:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0959999999999999e-05 2024-04-26T01:08:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187730+0000", 2024-04-26T01:08:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00126672 2024-04-26T01:08:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187762+0000", 2024-04-26T01:08:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.2759999999999998e-05 2024-04-26T01:08:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.203020+0000", 2024-04-26T01:08:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.015257409 2024-04-26T01:08:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.203093+0000", 2024-04-26T01:08:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2894999999999999e-05 2024-04-26T01:08:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:770 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~671744] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:08:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.264210+0000", 2024-04-26T01:08:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.251371198999999, 2024-04-26T01:08:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016944765000000001, 2024-04-26T01:08:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 770 2024-04-26T01:08:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.264210+0000", 2024-04-26T01:08:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.264210+0000", 2024-04-26T01:08:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.264214+0000", 2024-04-26T01:08:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.2799999999999999e-06 2024-04-26T01:08:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.265140+0000", 2024-04-26T01:08:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00092632999999999995 2024-04-26T01:08:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.265176+0000", 2024-04-26T01:08:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6253e-05 2024-04-26T01:08:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.265192+0000", 2024-04-26T01:08:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5792999999999999e-05 2024-04-26T01:08:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.265340+0000", 2024-04-26T01:08:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000148212 2024-04-26T01:08:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.265514+0000", 2024-04-26T01:08:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017378200000000001 2024-04-26T01:08:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.586 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:47.586 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.267348+0000", 2024-04-26T01:08:47.586 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0018339319999999999 2024-04-26T01:08:47.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.586 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.586 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:47.586 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.280762+0000", 2024-04-26T01:08:47.586 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.01341373 2024-04-26T01:08:47.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.586 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.586 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:47.586 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.280956+0000", 2024-04-26T01:08:47.586 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019426400000000001 2024-04-26T01:08:47.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.587 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.587 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.281155+0000", 2024-04-26T01:08:47.587 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019918900000000001 2024-04-26T01:08:47.587 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.587 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.587 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.587 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:774 3.0s0 3:0982979f:::smithi16428812-35:head [setxattr _header (57) in=7b] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:08:47.587 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.273352+0000", 2024-04-26T01:08:47.587 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.242229654999999, 2024-04-26T01:08:47.587 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.31518492300000001, 2024-04-26T01:08:47.588 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.588 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.588 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:47.588 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.588 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.588 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.588 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 774 2024-04-26T01:08:47.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.588 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.588 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.588 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.588 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273352+0000", 2024-04-26T01:08:47.588 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.589 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.589 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.589 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273352+0000", 2024-04-26T01:08:47.589 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.589 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.589 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.589 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273353+0000", 2024-04-26T01:08:47.589 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.407e-06 2024-04-26T01:08:47.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.589 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.589 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.589 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273361+0000", 2024-04-26T01:08:47.589 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.8960000000000003e-06 2024-04-26T01:08:47.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.590 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.590 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273382+0000", 2024-04-26T01:08:47.590 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0564000000000002e-05 2024-04-26T01:08:47.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.590 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.590 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273397+0000", 2024-04-26T01:08:47.590 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5492999999999999e-05 2024-04-26T01:08:47.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.590 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.590 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.282682+0000", 2024-04-26T01:08:47.590 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0092849830000000001 2024-04-26T01:08:47.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.591 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.591 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.282844+0000", 2024-04-26T01:08:47.591 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000161562 2024-04-26T01:08:47.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.591 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:47.591 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.283651+0000", 2024-04-26T01:08:47.591 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00080674999999999996 2024-04-26T01:08:47.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.591 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:47.591 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.285414+0000", 2024-04-26T01:08:47.592 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0017628800000000001 2024-04-26T01:08:47.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.592 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:47.592 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.588422+0000", 2024-04-26T01:08:47.592 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.30300855599999998 2024-04-26T01:08:47.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.592 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.592 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.588537+0000", 2024-04-26T01:08:47.592 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000114832 2024-04-26T01:08:47.592 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.592 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.592 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.593 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:775 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:08:47.593 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.273465+0000", 2024-04-26T01:08:47.593 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 10.242117135999999, 2024-04-26T01:08:47.593 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.31904869699999999, 2024-04-26T01:08:47.593 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.593 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.593 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:47.593 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.593 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.593 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.593 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 775 2024-04-26T01:08:47.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.594 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.594 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.594 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.594 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273465+0000", 2024-04-26T01:08:47.594 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.594 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.594 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.594 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273465+0000", 2024-04-26T01:08:47.594 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.594 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.594 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.594 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273466+0000", 2024-04-26T01:08:47.595 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4929999999999999e-06 2024-04-26T01:08:47.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.595 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.595 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273472+0000", 2024-04-26T01:08:47.595 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.2040000000000002e-06 2024-04-26T01:08:47.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.595 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.595 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273487+0000", 2024-04-26T01:08:47.595 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4486e-05 2024-04-26T01:08:47.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.595 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.596 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273497+0000", 2024-04-26T01:08:47.596 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0020000000000001e-05 2024-04-26T01:08:47.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.596 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.596 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.284233+0000", 2024-04-26T01:08:47.596 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010736104999999999 2024-04-26T01:08:47.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.596 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:47.596 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.284468+0000", 2024-04-26T01:08:47.596 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000235508 2024-04-26T01:08:47.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.597 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.597 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.589239+0000", 2024-04-26T01:08:47.597 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.30477114199999999 2024-04-26T01:08:47.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.597 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.597 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.589355+0000", 2024-04-26T01:08:47.597 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000115987 2024-04-26T01:08:47.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.597 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.597 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.592513+0000", 2024-04-26T01:08:47.597 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0031577520000000002 2024-04-26T01:08:47.597 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.598 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.598 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.598 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.598 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:797 3.0s0 3:0d858d01:::smithi16428812-41:head [append 0~720896] snapc 16=[16,15] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-26T01:08:47.598 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:39.267373+0000", 2024-04-26T01:08:47.598 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 8.2482084830000009, 2024-04-26T01:08:47.598 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.019993444999999999, 2024-04-26T01:08:47.598 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.598 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.598 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:47.598 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.598 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.598 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.599 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 797 2024-04-26T01:08:47.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.599 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.599 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.599 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.267373+0000", 2024-04-26T01:08:47.599 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.599 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.599 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.267373+0000", 2024-04-26T01:08:47.599 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.599 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.600 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.267377+0000", 2024-04-26T01:08:47.600 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7869999999999999e-06 2024-04-26T01:08:47.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.600 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.600 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.269934+0000", 2024-04-26T01:08:47.600 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0025572030000000001 2024-04-26T01:08:47.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.600 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.600 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.269981+0000", 2024-04-26T01:08:47.600 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.6409999999999998e-05 2024-04-26T01:08:47.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.601 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.601 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.270004+0000", 2024-04-26T01:08:47.601 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3563000000000001e-05 2024-04-26T01:08:47.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.601 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.601 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.270160+0000", 2024-04-26T01:08:47.601 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015629899999999999 2024-04-26T01:08:47.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.601 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.601 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.270398+0000", 2024-04-26T01:08:47.601 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000237915 2024-04-26T01:08:47.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.602 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:47.602 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.273343+0000", 2024-04-26T01:08:47.602 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0029442240000000001 2024-04-26T01:08:47.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.602 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:47.602 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.281421+0000", 2024-04-26T01:08:47.602 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0080788019999999995 2024-04-26T01:08:47.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.602 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:47.602 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.287233+0000", 2024-04-26T01:08:47.603 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0058111930000000001 2024-04-26T01:08:47.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.603 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.603 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.287367+0000", 2024-04-26T01:08:47.603 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013404899999999999 2024-04-26T01:08:47.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.603 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.603 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:798 3.0s0 3:0d858d01:::smithi16428812-41:head [append 0~622592] snapc 16=[16,15] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-26T01:08:47.603 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:39.272733+0000", 2024-04-26T01:08:47.603 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 8.2428481710000003, 2024-04-26T01:08:47.604 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014904333000000001, 2024-04-26T01:08:47.604 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.604 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.604 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:47.604 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.604 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.604 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.604 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 798 2024-04-26T01:08:47.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.604 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.604 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.604 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.604 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.272733+0000", 2024-04-26T01:08:47.604 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.605 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.605 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.272733+0000", 2024-04-26T01:08:47.605 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.605 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.605 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.272738+0000", 2024-04-26T01:08:47.605 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.1030000000000004e-06 2024-04-26T01:08:47.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.605 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.605 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.274541+0000", 2024-04-26T01:08:47.605 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001803303 2024-04-26T01:08:47.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.606 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.606 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.274582+0000", 2024-04-26T01:08:47.606 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.1393000000000002e-05 2024-04-26T01:08:47.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.606 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.606 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.274606+0000", 2024-04-26T01:08:47.606 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3292999999999999e-05 2024-04-26T01:08:47.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.606 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.606 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.276596+0000", 2024-04-26T01:08:47.607 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0019906669999999998 2024-04-26T01:08:47.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.607 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.607 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.276858+0000", 2024-04-26T01:08:47.607 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00026163100000000002 2024-04-26T01:08:47.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.607 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:47.607 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.279030+0000", 2024-04-26T01:08:47.607 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0021722360000000001 2024-04-26T01:08:47.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.607 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:47.607 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.281804+0000", 2024-04-26T01:08:47.608 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0027742180000000002 2024-04-26T01:08:47.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.608 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:47.608 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.287522+0000", 2024-04-26T01:08:47.608 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0057177720000000003 2024-04-26T01:08:47.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.608 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.608 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.287638+0000", 2024-04-26T01:08:47.608 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000115717 2024-04-26T01:08:47.608 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.608 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.608 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:801 3.0s0 3:0d858d01:::smithi16428812-41:head [setxattr _header (57) in=7b] snapc 16=[16,15] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-26T01:08:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:39.282568+0000", 2024-04-26T01:08:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 8.2330140660000009, 2024-04-26T01:08:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.018474469, 2024-04-26T01:08:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 801 2024-04-26T01:08:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.610 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.610 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.610 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282568+0000", 2024-04-26T01:08:47.610 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.610 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.610 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.610 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282568+0000", 2024-04-26T01:08:47.610 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.610 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.610 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.610 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282569+0000", 2024-04-26T01:08:47.610 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7719999999999999e-06 2024-04-26T01:08:47.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.611 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.611 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.611 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282580+0000", 2024-04-26T01:08:47.611 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0562e-05 2024-04-26T01:08:47.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.611 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.611 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.611 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282598+0000", 2024-04-26T01:08:47.611 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7960000000000001e-05 2024-04-26T01:08:47.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.611 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.611 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.611 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282612+0000", 2024-04-26T01:08:47.612 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3614e-05 2024-04-26T01:08:47.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.612 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.612 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.289835+0000", 2024-04-26T01:08:47.612 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0072236469999999997 2024-04-26T01:08:47.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.612 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.612 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.290022+0000", 2024-04-26T01:08:47.612 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00018657600000000001 2024-04-26T01:08:47.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.612 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:47.612 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.290813+0000", 2024-04-26T01:08:47.613 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00079124600000000001 2024-04-26T01:08:47.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.613 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:47.613 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.292480+0000", 2024-04-26T01:08:47.613 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001667237 2024-04-26T01:08:47.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.613 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:47.613 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.300923+0000", 2024-04-26T01:08:47.613 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0084426929999999994 2024-04-26T01:08:47.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.301042+0000", 2024-04-26T01:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000119162 2024-04-26T01:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:802 3.0s0 3:0d858d01:::smithi16428812-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-04-26T01:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:39.282634+0000", 2024-04-26T01:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 8.2329480860000004, 2024-04-26T01:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.025775884999999998, 2024-04-26T01:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 802 2024-04-26T01:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282634+0000", 2024-04-26T01:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282634+0000", 2024-04-26T01:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282635+0000", 2024-04-26T01:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2619999999999999e-06 2024-04-26T01:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282641+0000", 2024-04-26T01:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.2319999999999998e-06 2024-04-26T01:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282658+0000", 2024-04-26T01:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7274000000000001e-05 2024-04-26T01:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282670+0000", 2024-04-26T01:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1243000000000001e-05 2024-04-26T01:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.292820+0000", 2024-04-26T01:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010150899999999999 2024-04-26T01:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.293017+0000", 2024-04-26T01:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019624200000000001 2024-04-26T01:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.301690+0000", 2024-04-26T01:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0086736279999999992 2024-04-26T01:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:47.621 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.301945+0000", 2024-04-26T01:08:47.621 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000254885 2024-04-26T01:08:47.621 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:47.621 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:47.621 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:47.621 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.308409+0000", 2024-04-26T01:08:47.621 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0064642190000000002 2024-04-26T01:08:47.621 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.621 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.621 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.622 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:47.622 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:47.622 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:47.623 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:08:47.624 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:47.623+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:47.713 INFO:tasks.rados.rados.0.smithi164.stdout:349: done (8 left) 2024-04-26T01:08:47.713 INFO:tasks.rados.rados.0.smithi164.stdout:352: done (7 left) 2024-04-26T01:08:47.713 INFO:tasks.rados.rados.0.smithi164.stdout:353: done (6 left) 2024-04-26T01:08:47.713 INFO:tasks.rados.rados.0.smithi164.stdout:357: done (5 left) 2024-04-26T01:08:47.713 INFO:tasks.rados.rados.0.smithi164.stdout:358: rmattr oid 19 current snap is 26 2024-04-26T01:08:47.714 INFO:tasks.rados.rados.0.smithi164.stdout:358: done (5 left) 2024-04-26T01:08:47.714 INFO:tasks.rados.rados.0.smithi164.stdout:359: snap_create 2024-04-26T01:08:47.714 INFO:tasks.rados.rados.0.smithi164.stdout:356: finishing copy_from racing read to smithi16428812-45 2024-04-26T01:08:47.714 INFO:tasks.rados.rados.0.smithi164.stdout:356: finishing copy_from to smithi16428812-45 2024-04-26T01:08:47.714 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 45 v 47 (ObjNum 99 snap 11 seq_num 99) dirty exists 2024-04-26T01:08:47.724 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:47.720+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:47.744 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:47.744 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:47.745 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:08:47.745 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:47.759 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:08:47.760 INFO:tasks.rados.rados.0.smithi164.stdout:350: finishing rollback tid 2 to smithi16428812-15 2024-04-26T01:08:47.760 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 15 v 69 (ObjNum 118 snap 15 seq_num 118) dirty exists 2024-04-26T01:08:47.826 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:47.824+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:47.882 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:47.882 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:47.882 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2024-04-26T01:08:47.882 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2024-04-26T01:08:47.883 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:47.904 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:08:47.927 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:47.927+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:48.024 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:48.024 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2024-04-26T01:08:48.024 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2024-04-26T01:08:48.024 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2024-04-26T01:08:48.024 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.024 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:4 3.ds0 3:bcd9cb81:::smithi16428812-2:head [append 0~32768,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:48.024 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.302987+0000", 2024-04-26T01:08:48.024 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 48.719548248000002, 2024-04-26T01:08:48.024 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012685460000000001, 2024-04-26T01:08:48.025 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.025 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.025 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:48.025 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.025 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.025 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.025 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 4 2024-04-26T01:08:48.025 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.025 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.025 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.025 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.025 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.302987+0000", 2024-04-26T01:08:48.025 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.026 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.026 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.026 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.026 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.302987+0000", 2024-04-26T01:08:48.026 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.026 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.026 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.026 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.026 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.302989+0000", 2024-04-26T01:08:48.026 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.266e-06 2024-04-26T01:08:48.026 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.026 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.026 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.027 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303040+0000", 2024-04-26T01:08:48.027 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1029000000000002e-05 2024-04-26T01:08:48.027 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.027 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.027 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.027 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303059+0000", 2024-04-26T01:08:48.027 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8909000000000001e-05 2024-04-26T01:08:48.027 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.027 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.027 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.027 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303074+0000", 2024-04-26T01:08:48.027 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4812e-05 2024-04-26T01:08:48.027 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.028 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.028 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.028 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303197+0000", 2024-04-26T01:08:48.028 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000122748 2024-04-26T01:08:48.028 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.028 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.028 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.028 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303424+0000", 2024-04-26T01:08:48.028 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022748500000000001 2024-04-26T01:08:48.028 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.028 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.028 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.028 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304556+0000", 2024-04-26T01:08:48.028 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001131948 2024-04-26T01:08:48.029 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.029 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.029 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.029 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.313947+0000", 2024-04-26T01:08:48.029 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0093914069999999992 2024-04-26T01:08:48.029 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.029 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.029 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:48.029 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315465+0000", 2024-04-26T01:08:48.029 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001517673 2024-04-26T01:08:48.029 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.029 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.029 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.029 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315672+0000", 2024-04-26T01:08:48.030 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000207183 2024-04-26T01:08:48.030 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.030 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.030 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.030 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.030 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.030 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:5 3.ds0 3:bcd9cb81:::smithi16428812-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:48.030 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.303108+0000", 2024-04-26T01:08:48.030 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 48.719427250000003, 2024-04-26T01:08:48.030 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013204789, 2024-04-26T01:08:48.030 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.030 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.030 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:48.030 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.031 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.031 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.031 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 5 2024-04-26T01:08:48.031 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.031 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.031 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.031 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.031 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303108+0000", 2024-04-26T01:08:48.031 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.031 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.031 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.031 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.031 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303108+0000", 2024-04-26T01:08:48.031 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.032 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.032 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.032 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.032 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303109+0000", 2024-04-26T01:08:48.032 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3149999999999999e-06 2024-04-26T01:08:48.032 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.032 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.032 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.032 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303117+0000", 2024-04-26T01:08:48.032 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1899999999999995e-06 2024-04-26T01:08:48.032 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.032 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.032 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.032 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303131+0000", 2024-04-26T01:08:48.032 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3890000000000001e-05 2024-04-26T01:08:48.033 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.033 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.033 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.033 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303143+0000", 2024-04-26T01:08:48.033 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1773999999999999e-05 2024-04-26T01:08:48.033 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.033 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.033 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.033 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.305710+0000", 2024-04-26T01:08:48.033 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0025668930000000002 2024-04-26T01:08:48.033 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.033 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.033 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.033 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.305934+0000", 2024-04-26T01:08:48.034 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022386800000000001 2024-04-26T01:08:48.034 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.034 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.034 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.034 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.307007+0000", 2024-04-26T01:08:48.034 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010735930000000001 2024-04-26T01:08:48.034 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.034 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.034 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.034 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.313953+0000", 2024-04-26T01:08:48.034 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006945657 2024-04-26T01:08:48.034 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.034 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.034 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:48.035 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316089+0000", 2024-04-26T01:08:48.035 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021360509999999999 2024-04-26T01:08:48.035 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.035 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.035 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.035 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316313+0000", 2024-04-26T01:08:48.035 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000223558 2024-04-26T01:08:48.035 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.035 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.035 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.035 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.035 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.035 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:6 3.ds0 3:bcd9cb81:::smithi16428812-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:48.035 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.303162+0000", 2024-04-26T01:08:48.036 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 48.719373267999998, 2024-04-26T01:08:48.036 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.027157324, 2024-04-26T01:08:48.036 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.036 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.036 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:48.036 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.036 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.036 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.036 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 6 2024-04-26T01:08:48.036 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.036 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.036 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.036 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.036 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303162+0000", 2024-04-26T01:08:48.037 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.037 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.037 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.037 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.037 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303162+0000", 2024-04-26T01:08:48.037 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.037 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.037 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.037 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.037 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303163+0000", 2024-04-26T01:08:48.037 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.145e-06 2024-04-26T01:08:48.037 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.037 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.038 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.038 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303169+0000", 2024-04-26T01:08:48.038 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8150000000000002e-06 2024-04-26T01:08:48.038 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.038 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.038 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.038 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303184+0000", 2024-04-26T01:08:48.038 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5116000000000001e-05 2024-04-26T01:08:48.038 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.038 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.038 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.038 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303196+0000", 2024-04-26T01:08:48.038 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1778e-05 2024-04-26T01:08:48.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.039 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.039 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.039 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308497+0000", 2024-04-26T01:08:48.039 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0053009449999999996 2024-04-26T01:08:48.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.039 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.039 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:48.039 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308937+0000", 2024-04-26T01:08:48.039 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00044013100000000002 2024-04-26T01:08:48.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.039 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.039 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.039 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.317636+0000", 2024-04-26T01:08:48.040 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0086997570000000007 2024-04-26T01:08:48.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.040 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.040 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.040 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.317809+0000", 2024-04-26T01:08:48.040 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017204899999999999 2024-04-26T01:08:48.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.040 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.040 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.040 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.330319+0000", 2024-04-26T01:08:48.040 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012510588 2024-04-26T01:08:48.040 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.040 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.040 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.041 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.041 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.041 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:34 3.ds0 3:b1366e28:::smithi16428812-12:head [append 0~196608,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:48.041 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321139+0000", 2024-04-26T01:08:48.041 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 48.701396029999998, 2024-04-26T01:08:48.041 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015400954999999999, 2024-04-26T01:08:48.041 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.041 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.041 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:48.041 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.041 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.041 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.041 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 34 2024-04-26T01:08:48.042 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.042 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.042 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.042 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.042 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321139+0000", 2024-04-26T01:08:48.042 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.042 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.042 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.042 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.042 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321139+0000", 2024-04-26T01:08:48.042 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.042 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.042 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.042 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.043 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321142+0000", 2024-04-26T01:08:48.043 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8660000000000002e-06 2024-04-26T01:08:48.043 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.043 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.043 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.043 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321475+0000", 2024-04-26T01:08:48.043 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00033334599999999999 2024-04-26T01:08:48.043 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.043 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.043 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.043 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321504+0000", 2024-04-26T01:08:48.043 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8445e-05 2024-04-26T01:08:48.043 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.043 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.043 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.044 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321518+0000", 2024-04-26T01:08:48.044 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4708e-05 2024-04-26T01:08:48.044 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.044 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.044 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.044 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322532+0000", 2024-04-26T01:08:48.044 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001013246 2024-04-26T01:08:48.044 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.044 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.044 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.044 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322688+0000", 2024-04-26T01:08:48.044 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000156525 2024-04-26T01:08:48.044 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.044 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.045 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.045 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323904+0000", 2024-04-26T01:08:48.045 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012160160000000001 2024-04-26T01:08:48.045 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.045 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.045 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.045 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.332759+0000", 2024-04-26T01:08:48.045 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0088551199999999993 2024-04-26T01:08:48.045 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.045 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.045 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:48.045 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336395+0000", 2024-04-26T01:08:48.045 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0036352559999999999 2024-04-26T01:08:48.045 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.046 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.046 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.046 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336540+0000", 2024-04-26T01:08:48.046 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000145427 2024-04-26T01:08:48.046 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.046 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.046 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.046 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.046 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.046 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:35 3.ds0 3:b1366e28:::smithi16428812-12:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:48.046 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321544+0000", 2024-04-26T01:08:48.046 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 48.700990912000002, 2024-04-26T01:08:48.046 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015414796, 2024-04-26T01:08:48.047 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.047 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.047 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:48.047 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.047 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.047 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.047 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 35 2024-04-26T01:08:48.047 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.047 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.047 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.047 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.047 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321544+0000", 2024-04-26T01:08:48.047 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.047 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.048 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.048 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.048 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321544+0000", 2024-04-26T01:08:48.048 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.048 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.048 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.048 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.048 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321545+0000", 2024-04-26T01:08:48.048 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3340000000000001e-06 2024-04-26T01:08:48.048 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.048 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.048 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.048 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321553+0000", 2024-04-26T01:08:48.048 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4739999999999999e-06 2024-04-26T01:08:48.049 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.049 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.049 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.049 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321571+0000", 2024-04-26T01:08:48.049 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7688000000000001e-05 2024-04-26T01:08:48.049 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.049 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.049 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.049 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321614+0000", 2024-04-26T01:08:48.049 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3047000000000003e-05 2024-04-26T01:08:48.049 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.049 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.049 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.049 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324961+0000", 2024-04-26T01:08:48.049 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003347184 2024-04-26T01:08:48.050 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.050 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.050 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.050 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.325113+0000", 2024-04-26T01:08:48.050 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000152287 2024-04-26T01:08:48.050 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.050 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.050 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.050 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.327055+0000", 2024-04-26T01:08:48.050 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001942292 2024-04-26T01:08:48.050 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.050 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.050 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.050 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.332763+0000", 2024-04-26T01:08:48.051 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0057070819999999996 2024-04-26T01:08:48.051 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.051 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.051 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:48.051 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336824+0000", 2024-04-26T01:08:48.051 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0040615449999999997 2024-04-26T01:08:48.051 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.051 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.051 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.051 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336959+0000", 2024-04-26T01:08:48.051 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013486300000000001 2024-04-26T01:08:48.051 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.051 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.051 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.051 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.052 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.052 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:36 3.ds0 3:b1366e28:::smithi16428812-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:48.052 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321633+0000", 2024-04-26T01:08:48.052 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 48.700902075000002, 2024-04-26T01:08:48.052 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.019982366000000001, 2024-04-26T01:08:48.052 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.052 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.052 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:48.052 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.052 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.052 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.052 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 36 2024-04-26T01:08:48.052 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.053 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.053 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.053 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.053 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321633+0000", 2024-04-26T01:08:48.053 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.053 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.053 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.053 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.053 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321633+0000", 2024-04-26T01:08:48.053 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.053 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.053 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.053 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.053 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321634+0000", 2024-04-26T01:08:48.053 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.268e-06 2024-04-26T01:08:48.054 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.054 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.054 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.054 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321641+0000", 2024-04-26T01:08:48.054 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5100000000000004e-06 2024-04-26T01:08:48.054 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.054 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.054 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.054 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321656+0000", 2024-04-26T01:08:48.054 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5191000000000001e-05 2024-04-26T01:08:48.054 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.054 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.055 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.055 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321666+0000", 2024-04-26T01:08:48.055 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0178e-05 2024-04-26T01:08:48.055 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.055 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.055 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.055 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.333161+0000", 2024-04-26T01:08:48.055 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011494605999999999 2024-04-26T01:08:48.055 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.055 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.055 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:48.055 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.333311+0000", 2024-04-26T01:08:48.055 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015046100000000001 2024-04-26T01:08:48.055 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.056 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.056 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.056 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338017+0000", 2024-04-26T01:08:48.056 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0047055489999999998 2024-04-26T01:08:48.056 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.056 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.056 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.056 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338172+0000", 2024-04-26T01:08:48.056 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015476500000000001 2024-04-26T01:08:48.056 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.056 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.056 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.056 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341615+0000", 2024-04-26T01:08:48.056 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0034438379999999998 2024-04-26T01:08:48.056 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.057 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.057 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.057 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.057 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.057 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpReadReply(3.ds0 33/30 ECSubReadReply(tid=3, attrs_read=0))", 2024-04-26T01:08:48.057 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.322592+0000", 2024-04-26T01:08:48.057 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 48.699943064000003, 2024-04-26T01:08:48.057 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077557909999999997, 2024-04-26T01:08:48.057 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.057 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.057 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.057 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.057 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.057 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.058 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322592+0000", 2024-04-26T01:08:48.058 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.058 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.058 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.058 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.058 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322592+0000", 2024-04-26T01:08:48.058 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.058 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.058 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.058 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.058 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322594+0000", 2024-04-26T01:08:48.058 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7740000000000001e-06 2024-04-26T01:08:48.058 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.059 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.059 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.059 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322617+0000", 2024-04-26T01:08:48.059 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3074999999999998e-05 2024-04-26T01:08:48.059 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.059 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.059 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.059 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322695+0000", 2024-04-26T01:08:48.059 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7892999999999994e-05 2024-04-26T01:08:48.059 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.059 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.059 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.059 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322710+0000", 2024-04-26T01:08:48.059 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.505e-05 2024-04-26T01:08:48.059 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.060 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.060 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.060 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329473+0000", 2024-04-26T01:08:48.060 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006763682 2024-04-26T01:08:48.060 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.060 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.060 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.060 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.330348+0000", 2024-04-26T01:08:48.060 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000874317 2024-04-26T01:08:48.060 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.060 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.060 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.060 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.060 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.061 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 32/30 ECSubWrite(tid=0, reqid=client.4521.0:28, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:48.061 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.326088+0000", 2024-04-26T01:08:48.061 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 48.696447229, 2024-04-26T01:08:48.061 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0097951900000000005, 2024-04-26T01:08:48.061 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.061 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.061 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.061 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.061 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.061 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.061 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326088+0000", 2024-04-26T01:08:48.061 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.061 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.061 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.061 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.062 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326088+0000", 2024-04-26T01:08:48.062 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.062 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.062 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.062 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.062 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326089+0000", 2024-04-26T01:08:48.062 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5460000000000001e-06 2024-04-26T01:08:48.062 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.062 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.062 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.062 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326154+0000", 2024-04-26T01:08:48.062 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4979000000000002e-05 2024-04-26T01:08:48.062 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.062 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.063 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.063 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326251+0000", 2024-04-26T01:08:48.063 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7139999999999995e-05 2024-04-26T01:08:48.063 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.063 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.063 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.063 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326266+0000", 2024-04-26T01:08:48.063 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4355999999999999e-05 2024-04-26T01:08:48.063 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.063 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.063 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.063 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.330888+0000", 2024-04-26T01:08:48.063 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0046226310000000003 2024-04-26T01:08:48.064 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.064 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.064 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.064 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.331237+0000", 2024-04-26T01:08:48.064 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000348103 2024-04-26T01:08:48.064 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.064 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.064 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.064 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335801+0000", 2024-04-26T01:08:48.064 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0045642189999999996 2024-04-26T01:08:48.064 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.064 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.064 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.064 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335883+0000", 2024-04-26T01:08:48.065 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2216e-05 2024-04-26T01:08:48.065 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.065 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.065 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.065 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.065 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.065 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:46 3.ds0 3:b85e6c81:::smithi16428812-16:head [append 0~262144,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:48.065 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.338039+0000", 2024-04-26T01:08:48.065 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 48.684496140999997, 2024-04-26T01:08:48.065 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0092704039999999995, 2024-04-26T01:08:48.065 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.065 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.065 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:48.065 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.066 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.066 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.066 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 46 2024-04-26T01:08:48.066 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.066 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.066 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.066 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.066 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338039+0000", 2024-04-26T01:08:48.066 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.066 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.066 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.066 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.066 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338039+0000", 2024-04-26T01:08:48.067 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.067 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.067 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.067 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.067 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338041+0000", 2024-04-26T01:08:48.067 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4660000000000002e-06 2024-04-26T01:08:48.067 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.067 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.067 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.067 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338148+0000", 2024-04-26T01:08:48.067 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000106384 2024-04-26T01:08:48.067 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.067 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.067 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.067 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338170+0000", 2024-04-26T01:08:48.068 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2022e-05 2024-04-26T01:08:48.068 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.068 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.068 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.068 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338184+0000", 2024-04-26T01:08:48.068 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4345000000000001e-05 2024-04-26T01:08:48.068 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.068 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.068 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.068 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341687+0000", 2024-04-26T01:08:48.068 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003502631 2024-04-26T01:08:48.068 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.068 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.069 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.069 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341929+0000", 2024-04-26T01:08:48.069 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024248 2024-04-26T01:08:48.069 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.069 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.069 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.343657+0000", 2024-04-26T01:08:48.069 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001727302 2024-04-26T01:08:48.069 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.069 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.069 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346591+0000", 2024-04-26T01:08:48.069 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0029344420000000002 2024-04-26T01:08:48.070 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.070 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.070 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:48.070 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.347148+0000", 2024-04-26T01:08:48.070 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000556801 2024-04-26T01:08:48.070 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.070 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.070 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.070 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.347309+0000", 2024-04-26T01:08:48.070 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000161531 2024-04-26T01:08:48.070 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.070 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.070 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.070 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.071 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.071 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:47 3.ds0 3:b85e6c81:::smithi16428812-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:48.071 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.338373+0000", 2024-04-26T01:08:48.071 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 48.684162459, 2024-04-26T01:08:48.071 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012969223, 2024-04-26T01:08:48.071 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.071 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.071 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:48.071 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.071 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.071 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.071 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 47 2024-04-26T01:08:48.071 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.071 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.072 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.072 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.072 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338373+0000", 2024-04-26T01:08:48.072 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.072 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.072 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.072 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.072 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338373+0000", 2024-04-26T01:08:48.072 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.072 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.072 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.072 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.072 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338374+0000", 2024-04-26T01:08:48.073 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.782e-06 2024-04-26T01:08:48.073 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.073 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.073 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.073 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338383+0000", 2024-04-26T01:08:48.073 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.6570000000000003e-06 2024-04-26T01:08:48.073 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.073 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.073 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.073 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338402+0000", 2024-04-26T01:08:48.073 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8708000000000001e-05 2024-04-26T01:08:48.073 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.073 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.073 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.074 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338415+0000", 2024-04-26T01:08:48.074 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2976000000000001e-05 2024-04-26T01:08:48.074 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.074 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.074 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.074 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.344766+0000", 2024-04-26T01:08:48.074 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0063516110000000001 2024-04-26T01:08:48.074 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.074 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.074 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.074 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.344969+0000", 2024-04-26T01:08:48.074 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00020233 2024-04-26T01:08:48.074 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.074 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.075 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.075 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.345881+0000", 2024-04-26T01:08:48.075 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00091244999999999998 2024-04-26T01:08:48.075 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.075 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.075 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.075 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.347458+0000", 2024-04-26T01:08:48.075 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001576945 2024-04-26T01:08:48.075 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.075 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.075 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:48.075 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.351176+0000", 2024-04-26T01:08:48.075 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0037184449999999999 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.351342+0000", 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016531900000000001 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:48 3.ds0 3:b85e6c81:::smithi16428812-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.338562+0000", 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 48.683972928999999, 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.028604862000000002, 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.076 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.077 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:48.077 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.077 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.077 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.077 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 48 2024-04-26T01:08:48.077 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.077 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.077 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.077 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.077 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338562+0000", 2024-04-26T01:08:48.077 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.077 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.077 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.077 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.077 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338562+0000", 2024-04-26T01:08:48.078 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.078 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.078 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.078 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.078 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338564+0000", 2024-04-26T01:08:48.078 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5319999999999999e-06 2024-04-26T01:08:48.078 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.078 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.078 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.078 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338572+0000", 2024-04-26T01:08:48.078 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1780000000000003e-06 2024-04-26T01:08:48.078 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.078 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.078 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.079 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338587+0000", 2024-04-26T01:08:48.079 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5293000000000001e-05 2024-04-26T01:08:48.079 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.079 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.079 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.079 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338599+0000", 2024-04-26T01:08:48.079 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1968e-05 2024-04-26T01:08:48.079 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.079 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.079 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.079 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346313+0000", 2024-04-26T01:08:48.079 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077139060000000004 2024-04-26T01:08:48.079 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.079 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.079 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:48.080 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346451+0000", 2024-04-26T01:08:48.080 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013819400000000001 2024-04-26T01:08:48.080 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.080 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.080 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.080 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.352103+0000", 2024-04-26T01:08:48.080 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056514249999999999 2024-04-26T01:08:48.080 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.080 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.080 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.080 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.352448+0000", 2024-04-26T01:08:48.080 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034553299999999997 2024-04-26T01:08:48.080 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.080 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.081 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.081 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.367167+0000", 2024-04-26T01:08:48.081 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.014718833000000001 2024-04-26T01:08:48.081 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.081 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.081 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.081 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.081 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.081 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/30 ECSubWrite(tid=2, reqid=client.4521.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:48.081 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.339308+0000", 2024-04-26T01:08:48.081 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 48.683226785999999, 2024-04-26T01:08:48.081 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0072084560000000002, 2024-04-26T01:08:48.081 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.081 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.081 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.082 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.082 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.082 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.082 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339308+0000", 2024-04-26T01:08:48.082 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.082 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.082 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.082 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.082 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339308+0000", 2024-04-26T01:08:48.082 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.082 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.082 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.082 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.083 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339310+0000", 2024-04-26T01:08:48.083 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4959999999999999e-06 2024-04-26T01:08:48.083 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.083 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.083 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.083 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339328+0000", 2024-04-26T01:08:48.083 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8349999999999999e-05 2024-04-26T01:08:48.083 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.083 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.083 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.083 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339614+0000", 2024-04-26T01:08:48.083 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028623 2024-04-26T01:08:48.083 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.083 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.084 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.084 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339629+0000", 2024-04-26T01:08:48.084 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4803999999999999e-05 2024-04-26T01:08:48.084 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.084 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.084 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.084 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.340313+0000", 2024-04-26T01:08:48.084 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00068382200000000005 2024-04-26T01:08:48.084 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.084 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.084 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.084 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.340400+0000", 2024-04-26T01:08:48.084 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.7321e-05 2024-04-26T01:08:48.085 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.085 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.085 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.085 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346462+0000", 2024-04-26T01:08:48.085 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0060617939999999997 2024-04-26T01:08:48.085 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.085 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.085 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.085 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346517+0000", 2024-04-26T01:08:48.085 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4639000000000003e-05 2024-04-26T01:08:48.085 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.085 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.085 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.085 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.086 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.086 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/30 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=38'19, trim_to=0'0, roll_forward_to=37'17))", 2024-04-26T01:08:48.086 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:04.427965+0000", 2024-04-26T01:08:48.086 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 43.594570242000003, 2024-04-26T01:08:48.086 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0080739560000000002, 2024-04-26T01:08:48.086 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.086 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.086 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.086 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.086 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.086 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.086 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.427965+0000", 2024-04-26T01:08:48.086 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.086 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.087 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.087 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.087 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.427965+0000", 2024-04-26T01:08:48.087 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.087 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.087 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.087 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.087 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.427966+0000", 2024-04-26T01:08:48.087 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3600000000000003e-07 2024-04-26T01:08:48.087 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.087 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.087 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.088 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.428022+0000", 2024-04-26T01:08:48.088 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6805999999999998e-05 2024-04-26T01:08:48.088 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.088 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.088 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.088 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.428186+0000", 2024-04-26T01:08:48.088 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000163287 2024-04-26T01:08:48.088 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.088 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.088 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.089 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.428212+0000", 2024-04-26T01:08:48.089 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6163e-05 2024-04-26T01:08:48.089 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.089 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.089 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.089 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.430074+0000", 2024-04-26T01:08:48.089 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0018619369999999999 2024-04-26T01:08:48.089 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.089 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.089 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.089 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.430098+0000", 2024-04-26T01:08:48.089 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.393e-05 2024-04-26T01:08:48.089 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.089 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.089 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.090 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.435960+0000", 2024-04-26T01:08:48.090 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0058619930000000002 2024-04-26T01:08:48.090 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.090 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.090 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.090 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.436039+0000", 2024-04-26T01:08:48.090 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9103999999999993e-05 2024-04-26T01:08:48.090 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.090 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.090 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.090 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.090 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.090 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 60/30 ECSubWrite(tid=51, reqid=osd.15.0:51, at_version=60'45, trim_to=0'0, roll_forward_to=59'43))", 2024-04-26T01:08:48.090 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:08:48.091 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 21.850535440000002, 2024-04-26T01:08:48.091 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.15434805400000001, 2024-04-26T01:08:48.091 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.091 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.091 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.091 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.091 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.091 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.091 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:08:48.091 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.091 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.091 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.091 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.091 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:08:48.092 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.092 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.092 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.092 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.092 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:08:48.092 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7800000000000001e-07 2024-04-26T01:08:48.092 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.092 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.092 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.092 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172035+0000", 2024-04-26T01:08:48.092 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.5151999999999998e-05 2024-04-26T01:08:48.092 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.092 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.092 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.092 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324571+0000", 2024-04-26T01:08:48.093 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.152535484 2024-04-26T01:08:48.093 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.093 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.093 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.093 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324589+0000", 2024-04-26T01:08:48.093 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8323999999999999e-05 2024-04-26T01:08:48.093 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.093 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.093 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.093 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324739+0000", 2024-04-26T01:08:48.093 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014940099999999999 2024-04-26T01:08:48.093 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.093 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.093 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.094 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324758+0000", 2024-04-26T01:08:48.094 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9562999999999999e-05 2024-04-26T01:08:48.094 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.094 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.094 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.094 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.326281+0000", 2024-04-26T01:08:48.094 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001523349 2024-04-26T01:08:48.094 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.094 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.094 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.094 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.326348+0000", 2024-04-26T01:08:48.094 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6203000000000003e-05 2024-04-26T01:08:48.095 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.095 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.095 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.095 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.095 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.095 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.bs5 60/30 ECSubWrite(tid=349, reqid=osd.4.0:349, at_version=60'52, trim_to=0'0, roll_forward_to=59'50))", 2024-04-26T01:08:48.095 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:08:48.095 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 21.844958542000001, 2024-04-26T01:08:48.095 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.007081085, 2024-04-26T01:08:48.095 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.095 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.095 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.095 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.095 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.096 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.096 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:08:48.096 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.096 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.096 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.096 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:08:48.096 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.096 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.096 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.096 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:08:48.096 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1799999999999999e-07 2024-04-26T01:08:48.096 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.097 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.097 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.097 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177638+0000", 2024-04-26T01:08:48.097 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1736000000000007e-05 2024-04-26T01:08:48.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.097 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.097 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.097 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177710+0000", 2024-04-26T01:08:48.097 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1756999999999998e-05 2024-04-26T01:08:48.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.097 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.097 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.097 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177722+0000", 2024-04-26T01:08:48.097 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1516e-05 2024-04-26T01:08:48.098 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.098 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.098 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.098 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177868+0000", 2024-04-26T01:08:48.098 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000146112 2024-04-26T01:08:48.098 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.098 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.098 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.098 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177896+0000", 2024-04-26T01:08:48.098 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7943999999999999e-05 2024-04-26T01:08:48.098 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.098 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.098 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.098 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.184603+0000", 2024-04-26T01:08:48.099 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006707691 2024-04-26T01:08:48.099 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.099 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.099 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.099 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.184658+0000", 2024-04-26T01:08:48.099 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4110999999999997e-05 2024-04-26T01:08:48.099 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.099 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.099 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.099 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.099 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.099 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.bs5 60/30 ECSubWrite(tid=351, reqid=osd.4.0:351, at_version=60'54, trim_to=0'0, roll_forward_to=59'50))", 2024-04-26T01:08:48.099 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:26.178472+0000", 2024-04-26T01:08:48.100 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 21.844062870999998, 2024-04-26T01:08:48.100 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006950861, 2024-04-26T01:08:48.100 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.100 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.100 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.100 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.100 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.100 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.100 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178472+0000", 2024-04-26T01:08:48.100 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.100 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.100 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.100 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.100 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178472+0000", 2024-04-26T01:08:48.101 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.101 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.101 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.101 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.101 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178474+0000", 2024-04-26T01:08:48.101 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2199999999999999e-06 2024-04-26T01:08:48.101 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.101 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.101 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.101 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178532+0000", 2024-04-26T01:08:48.101 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7429000000000002e-05 2024-04-26T01:08:48.101 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.101 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.101 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.101 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178615+0000", 2024-04-26T01:08:48.102 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.3566000000000006e-05 2024-04-26T01:08:48.102 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.102 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.102 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.102 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178639+0000", 2024-04-26T01:08:48.102 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3736000000000001e-05 2024-04-26T01:08:48.102 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.102 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.102 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.102 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178895+0000", 2024-04-26T01:08:48.102 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00025574300000000001 2024-04-26T01:08:48.102 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.102 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.103 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.103 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.178920+0000", 2024-04-26T01:08:48.103 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5126999999999998e-05 2024-04-26T01:08:48.103 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.103 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.103 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.103 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.185333+0000", 2024-04-26T01:08:48.103 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0064127389999999998 2024-04-26T01:08:48.103 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.103 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.103 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.103 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.185423+0000", 2024-04-26T01:08:48.103 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0301000000000002e-05 2024-04-26T01:08:48.103 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.104 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.104 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.104 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.104 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.104 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:780 3.ds0 3:bcd9cb81:::smithi16428812-2:head [delete] snapc 15=[15] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:08:48.104 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:38.244242+0000", 2024-04-26T01:08:48.104 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.7782933229999998, 2024-04-26T01:08:48.104 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0084976549999999998, 2024-04-26T01:08:48.104 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.104 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.104 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:48.104 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.104 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.104 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.105 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 780 2024-04-26T01:08:48.105 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.105 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.105 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.105 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.105 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244242+0000", 2024-04-26T01:08:48.105 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.105 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.105 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.105 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.105 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244242+0000", 2024-04-26T01:08:48.105 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.106 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.106 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.106 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.106 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244246+0000", 2024-04-26T01:08:48.106 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8260000000000003e-06 2024-04-26T01:08:48.106 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.106 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.106 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.106 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244264+0000", 2024-04-26T01:08:48.106 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8209000000000001e-05 2024-04-26T01:08:48.106 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.106 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.106 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.107 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244294+0000", 2024-04-26T01:08:48.107 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0099000000000001e-05 2024-04-26T01:08:48.107 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.107 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.107 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.107 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244310+0000", 2024-04-26T01:08:48.107 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6336000000000001e-05 2024-04-26T01:08:48.107 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.107 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.107 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.107 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244466+0000", 2024-04-26T01:08:48.107 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015627100000000001 2024-04-26T01:08:48.107 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.107 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.107 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2024-04-26T01:08:48.108 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244486+0000", 2024-04-26T01:08:48.108 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9678999999999999e-05 2024-04-26T01:08:48.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.108 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.108 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.108 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.246633+0000", 2024-04-26T01:08:48.108 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021467359999999998 2024-04-26T01:08:48.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.108 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.108 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.108 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.246783+0000", 2024-04-26T01:08:48.108 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015011500000000001 2024-04-26T01:08:48.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.109 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.109 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.109 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.248075+0000", 2024-04-26T01:08:48.109 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012918890000000001 2024-04-26T01:08:48.109 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.109 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.109 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.109 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.249771+0000", 2024-04-26T01:08:48.109 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0016958209999999999 2024-04-26T01:08:48.109 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.109 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.109 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:48.109 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.252619+0000", 2024-04-26T01:08:48.109 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002848214 2024-04-26T01:08:48.110 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.110 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.110 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.110 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.252739+0000", 2024-04-26T01:08:48.110 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012046 2024-04-26T01:08:48.110 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.110 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.110 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.110 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.110 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.110 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.3s1 72/30 ECSubWrite(tid=240, reqid=client.4521.0:781, at_version=72'20, trim_to=0'0, roll_forward_to=63'18))", 2024-04-26T01:08:48.110 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:38.263581+0000", 2024-04-26T01:08:48.110 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.7589537550000003, 2024-04-26T01:08:48.110 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0089752990000000008, 2024-04-26T01:08:48.111 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.111 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.111 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.111 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.111 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.111 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.111 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.263581+0000", 2024-04-26T01:08:48.111 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.111 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.111 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.111 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.111 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.263581+0000", 2024-04-26T01:08:48.111 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.111 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.112 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.112 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.112 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.263583+0000", 2024-04-26T01:08:48.112 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.39e-06 2024-04-26T01:08:48.112 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.112 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.112 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.112 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.263949+0000", 2024-04-26T01:08:48.112 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00036607399999999998 2024-04-26T01:08:48.112 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.112 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.112 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.112 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.264035+0000", 2024-04-26T01:08:48.112 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.6055000000000002e-05 2024-04-26T01:08:48.112 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.113 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.113 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.113 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.264059+0000", 2024-04-26T01:08:48.113 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4071999999999999e-05 2024-04-26T01:08:48.113 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.113 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.113 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.113 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.264254+0000", 2024-04-26T01:08:48.113 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019476199999999999 2024-04-26T01:08:48.113 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.113 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.113 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.113 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.264282+0000", 2024-04-26T01:08:48.113 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8768999999999998e-05 2024-04-26T01:08:48.114 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.114 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.114 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.114 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.272454+0000", 2024-04-26T01:08:48.114 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0081717030000000006 2024-04-26T01:08:48.114 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.114 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.114 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.114 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.272557+0000", 2024-04-26T01:08:48.114 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010247399999999999 2024-04-26T01:08:48.114 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.114 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.114 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.114 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 72/30 ECSubWrite(tid=56, reqid=client.4521.0:785, at_version=72'49, trim_to=0'0, roll_forward_to=72'47))", 2024-04-26T01:08:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:38.303386+0000", 2024-04-26T01:08:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 9.7191494029999994, 2024-04-26T01:08:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0094813009999999993, 2024-04-26T01:08:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.303386+0000", 2024-04-26T01:08:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.116 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.116 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.116 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.116 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.303386+0000", 2024-04-26T01:08:48.116 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.116 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.116 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.116 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.116 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.303388+0000", 2024-04-26T01:08:48.116 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8439999999999998e-06 2024-04-26T01:08:48.116 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.116 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.116 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.116 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.304973+0000", 2024-04-26T01:08:48.116 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0015846090000000001 2024-04-26T01:08:48.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.117 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.117 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.305084+0000", 2024-04-26T01:08:48.117 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011120099999999999 2024-04-26T01:08:48.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.117 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.117 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.305116+0000", 2024-04-26T01:08:48.117 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1341999999999999e-05 2024-04-26T01:08:48.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.117 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.117 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.305312+0000", 2024-04-26T01:08:48.118 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000196286 2024-04-26T01:08:48.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.118 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.118 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.305347+0000", 2024-04-26T01:08:48.118 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.5185000000000003e-05 2024-04-26T01:08:48.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.118 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.118 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.312795+0000", 2024-04-26T01:08:48.118 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0074474119999999996 2024-04-26T01:08:48.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.118 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.119 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.312867+0000", 2024-04-26T01:08:48.119 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2422000000000006e-05 2024-04-26T01:08:48.119 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.119 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.119 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.119 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 81/30 ECSubWrite(tid=67, reqid=osd.15.0:67, at_version=81'54, trim_to=0'0, roll_forward_to=74'50))", 2024-04-26T01:08:48.119 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:47.314584+0000", 2024-04-26T01:08:48.119 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 0.70795078700000003, 2024-04-26T01:08:48.119 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0072634040000000002, 2024-04-26T01:08:48.119 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.119 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.119 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.119 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.120 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.120 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314584+0000", 2024-04-26T01:08:48.120 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.120 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.120 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314584+0000", 2024-04-26T01:08:48.120 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.120 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.120 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314586+0000", 2024-04-26T01:08:48.120 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2729999999999999e-06 2024-04-26T01:08:48.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.121 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.121 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314621+0000", 2024-04-26T01:08:48.121 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.5129999999999997e-05 2024-04-26T01:08:48.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.121 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.121 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314685+0000", 2024-04-26T01:08:48.121 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4212000000000001e-05 2024-04-26T01:08:48.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.121 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.121 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314699+0000", 2024-04-26T01:08:48.121 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4453e-05 2024-04-26T01:08:48.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.122 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.122 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.316889+0000", 2024-04-26T01:08:48.122 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002189954 2024-04-26T01:08:48.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.122 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.122 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.316916+0000", 2024-04-26T01:08:48.122 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6441999999999999e-05 2024-04-26T01:08:48.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.122 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.122 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.321756+0000", 2024-04-26T01:08:48.123 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0048401130000000001 2024-04-26T01:08:48.123 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.123 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.123 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.321848+0000", 2024-04-26T01:08:48.123 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1827e-05 2024-04-26T01:08:48.123 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.123 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.123 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.123 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.123 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.123 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:48.125 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:08:48.125 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:48.125+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:48.226 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:48.224+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:48.264 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:48.264 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:48.264 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:08:48.264 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:48.278 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:08:48.307 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-04-26T01:08:48.312 INFO:tasks.rados.rados.0.smithi164.stdout:350: done (5 left) 2024-04-26T01:08:48.312 INFO:tasks.rados.rados.0.smithi164.stdout:356: done (4 left) 2024-04-26T01:08:48.312 INFO:tasks.rados.rados.0.smithi164.stdout:359: done (3 left) 2024-04-26T01:08:48.312 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 45 current snap is 27 2024-04-26T01:08:48.312 INFO:tasks.rados.rados.0.smithi164.stdout:360: seq_num 150 ranges {1671168=114688} 2024-04-26T01:08:48.325 INFO:tasks.rados.rados.0.smithi164.stdout:360: writing smithi16428812-45 from 1671168 to 1785856 tid 1 2024-04-26T01:08:48.325 INFO:tasks.rados.rados.0.smithi164.stdout:361: copy_from oid 40 from oid 16 current snap is 27 2024-04-26T01:08:48.325 INFO:tasks.rados.rados.0.smithi164.stdout:362: rollback oid 33 current snap is 27 2024-04-26T01:08:48.325 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 33 to 24 2024-04-26T01:08:48.326 INFO:tasks.rados.rados.0.smithi164.stdout:363: snap_remove snap 23 2024-04-26T01:08:48.326 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:48.327+0000 7ff2871de640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 32810) UID: 0 2024-04-26T01:08:48.422 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:48.422 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:48.422 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2024-04-26T01:08:48.422 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2024-04-26T01:08:48.422 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:48.426 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:48.427 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:48.427+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:48.432 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T01:08:48.433 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T01:08:48.438 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:08:48.528 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:48.527+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:48.620 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:48.620 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2024-04-26T01:08:48.621 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2024-04-26T01:08:48.621 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2024-04-26T01:08:48.621 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.621 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:29 3.5s0 3:acf4d671:::smithi16428812-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:48.621 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321678+0000", 2024-04-26T01:08:48.621 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 49.298813641999999, 2024-04-26T01:08:48.621 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.016320530999999999, 2024-04-26T01:08:48.621 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.621 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.621 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:48.621 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.621 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.621 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.622 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 29 2024-04-26T01:08:48.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.622 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.622 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.622 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321678+0000", 2024-04-26T01:08:48.622 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.622 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.622 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321678+0000", 2024-04-26T01:08:48.622 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.623 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.623 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321680+0000", 2024-04-26T01:08:48.623 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1680000000000001e-06 2024-04-26T01:08:48.623 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.623 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.623 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.623 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321686+0000", 2024-04-26T01:08:48.623 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9539999999999998e-06 2024-04-26T01:08:48.623 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.623 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.623 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.623 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321702+0000", 2024-04-26T01:08:48.623 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5241e-05 2024-04-26T01:08:48.623 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.624 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.624 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321716+0000", 2024-04-26T01:08:48.624 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4384e-05 2024-04-26T01:08:48.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.624 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.624 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329428+0000", 2024-04-26T01:08:48.624 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077114369999999998 2024-04-26T01:08:48.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.624 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.624 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329669+0000", 2024-04-26T01:08:48.624 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000241561 2024-04-26T01:08:48.625 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.625 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.625 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.625 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.331103+0000", 2024-04-26T01:08:48.625 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001433871 2024-04-26T01:08:48.625 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.625 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.625 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.625 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336734+0000", 2024-04-26T01:08:48.625 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.005631306 2024-04-26T01:08:48.625 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.625 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.625 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:48.625 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337654+0000", 2024-04-26T01:08:48.625 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000919716 2024-04-26T01:08:48.626 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.626 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.626 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.626 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337999+0000", 2024-04-26T01:08:48.626 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034489299999999998 2024-04-26T01:08:48.626 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.626 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.626 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.626 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.626 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.626 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:30 3.5s0 3:acf4d671:::smithi16428812-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:48.626 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.322117+0000", 2024-04-26T01:08:48.626 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 49.298374586999998, 2024-04-26T01:08:48.626 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.021239319999999999, 2024-04-26T01:08:48.627 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.627 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.627 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:48.627 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.627 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.627 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.627 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 30 2024-04-26T01:08:48.627 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.627 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.627 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.627 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322117+0000", 2024-04-26T01:08:48.627 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.627 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.628 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.628 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322117+0000", 2024-04-26T01:08:48.628 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.628 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.628 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.628 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.628 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322119+0000", 2024-04-26T01:08:48.628 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3740000000000001e-06 2024-04-26T01:08:48.628 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.628 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.628 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.628 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322127+0000", 2024-04-26T01:08:48.628 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.994e-06 2024-04-26T01:08:48.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.629 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.629 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322146+0000", 2024-04-26T01:08:48.629 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9205999999999999e-05 2024-04-26T01:08:48.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.629 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.629 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322159+0000", 2024-04-26T01:08:48.629 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3377000000000001e-05 2024-04-26T01:08:48.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.630 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.630 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.334766+0000", 2024-04-26T01:08:48.630 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012606732000000001 2024-04-26T01:08:48.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.630 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:48.630 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335000+0000", 2024-04-26T01:08:48.630 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00023435799999999999 2024-04-26T01:08:48.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.630 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.630 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338932+0000", 2024-04-26T01:08:48.630 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0039317500000000003 2024-04-26T01:08:48.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.631 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.631 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339070+0000", 2024-04-26T01:08:48.631 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000137315 2024-04-26T01:08:48.631 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.631 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.631 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.631 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.343357+0000", 2024-04-26T01:08:48.631 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042872140000000001 2024-04-26T01:08:48.631 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.631 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.631 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.631 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.631 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.631 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:128 3.5s0 3:a661dd24:::smithi16428812-43:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:48.631 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:48.632 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 49.164006464000003, 2024-04-26T01:08:48.632 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.083673129999999998, 2024-04-26T01:08:48.632 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.632 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.632 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:48.632 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.632 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.632 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.632 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 128 2024-04-26T01:08:48.632 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.632 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.632 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.632 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.632 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:48.632 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.633 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.633 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.633 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.633 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:48.633 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.633 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.633 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.633 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.633 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:48.633 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1299999999999999e-07 2024-04-26T01:08:48.633 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.633 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.633 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.634 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456490+0000", 2024-04-26T01:08:48.634 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4029999999999999e-06 2024-04-26T01:08:48.634 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.634 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.634 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.634 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456501+0000", 2024-04-26T01:08:48.634 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1124999999999999e-05 2024-04-26T01:08:48.634 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.634 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.634 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.634 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456507+0000", 2024-04-26T01:08:48.634 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3180000000000003e-06 2024-04-26T01:08:48.634 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.634 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.635 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.635 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.458775+0000", 2024-04-26T01:08:48.635 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002268061 2024-04-26T01:08:48.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.635 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.635 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.635 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.458948+0000", 2024-04-26T01:08:48.635 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017263900000000001 2024-04-26T01:08:48.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.635 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.635 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.635 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.459498+0000", 2024-04-26T01:08:48.635 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00055037900000000004 2024-04-26T01:08:48.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.635 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.636 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.636 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460166+0000", 2024-04-26T01:08:48.636 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00066757899999999996 2024-04-26T01:08:48.636 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.636 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.636 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:48.636 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540012+0000", 2024-04-26T01:08:48.636 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.079846146000000007 2024-04-26T01:08:48.636 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.636 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.636 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.636 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540159+0000", 2024-04-26T01:08:48.636 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014676700000000001 2024-04-26T01:08:48.637 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.637 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.637 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.637 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.637 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.637 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:129 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:48.637 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:48.637 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 49.163977492999997, 2024-04-26T01:08:48.637 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.09570555, 2024-04-26T01:08:48.637 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.637 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.637 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:48.637 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.637 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.637 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.638 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 129 2024-04-26T01:08:48.638 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.638 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.638 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.638 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:48.638 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.638 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.638 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.638 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:48.638 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.638 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.639 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.639 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:48.639 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0699999999999997e-07 2024-04-26T01:08:48.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.639 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.639 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.639 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456518+0000", 2024-04-26T01:08:48.639 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.898e-06 2024-04-26T01:08:48.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.639 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.639 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.639 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456532+0000", 2024-04-26T01:08:48.639 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3597e-05 2024-04-26T01:08:48.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.640 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456537+0000", 2024-04-26T01:08:48.640 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.242e-06 2024-04-26T01:08:48.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.640 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.459839+0000", 2024-04-26T01:08:48.640 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033019830000000001 2024-04-26T01:08:48.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:48.640 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.459972+0000", 2024-04-26T01:08:48.640 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000133199 2024-04-26T01:08:48.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.641 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.641 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.641 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540208+0000", 2024-04-26T01:08:48.641 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.080236319 2024-04-26T01:08:48.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.641 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.641 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.641 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540275+0000", 2024-04-26T01:08:48.641 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6486e-05 2024-04-26T01:08:48.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.641 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.641 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.641 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.552220+0000", 2024-04-26T01:08:48.641 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011945318999999999 2024-04-26T01:08:48.641 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.642 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.642 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.642 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 33/30 ECSubWriteReply(tid=9, last_complete=33'6, committed=1, applied=1))", 2024-04-26T01:08:48.642 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:48.642 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 49.160196419000002, 2024-04-26T01:08:48.642 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.079879818000000005, 2024-04-26T01:08:48.642 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.642 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.642 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.642 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.642 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.643 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:48.643 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.643 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.643 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:48.643 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.643 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.643 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:48.643 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6599999999999996e-07 2024-04-26T01:08:48.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.643 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.644 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460301+0000", 2024-04-26T01:08:48.644 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4689999999999999e-06 2024-04-26T01:08:48.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.644 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.644 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.539830+0000", 2024-04-26T01:08:48.644 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.079528926999999999 2024-04-26T01:08:48.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.644 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.644 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.539834+0000", 2024-04-26T01:08:48.644 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5430000000000004e-06 2024-04-26T01:08:48.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.644 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.645 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.539887+0000", 2024-04-26T01:08:48.645 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2998999999999999e-05 2024-04-26T01:08:48.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.645 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.645 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540175+0000", 2024-04-26T01:08:48.645 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028821400000000002 2024-04-26T01:08:48.645 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.645 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.645 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.645 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:183 3.5s0 3:a8ed6f44:::smithi16428812-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:48.645 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:48.645 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 49.085298430999998, 2024-04-26T01:08:48.646 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.017578818999999999, 2024-04-26T01:08:48.646 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.646 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.646 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:48.646 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.646 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.646 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.647 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 183 2024-04-26T01:08:48.647 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.647 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.647 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.647 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.648 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:48.648 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.648 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.648 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.648 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.648 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:48.648 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.648 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.648 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.648 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.648 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:48.648 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9500000000000002e-07 2024-04-26T01:08:48.648 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.648 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.648 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.649 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535198+0000", 2024-04-26T01:08:48.649 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6430000000000001e-06 2024-04-26T01:08:48.649 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.649 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.649 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.649 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535207+0000", 2024-04-26T01:08:48.649 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9730000000000003e-06 2024-04-26T01:08:48.649 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.649 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.649 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.650 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535215+0000", 2024-04-26T01:08:48.650 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1540000000000002e-06 2024-04-26T01:08:48.650 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.650 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.650 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.650 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.537794+0000", 2024-04-26T01:08:48.650 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0025788740000000001 2024-04-26T01:08:48.650 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.650 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.650 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:48.650 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.537868+0000", 2024-04-26T01:08:48.650 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4028000000000003e-05 2024-04-26T01:08:48.651 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.651 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.651 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.550967+0000", 2024-04-26T01:08:48.651 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013099006999999999 2024-04-26T01:08:48.651 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.651 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.651 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551052+0000", 2024-04-26T01:08:48.651 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4667999999999996e-05 2024-04-26T01:08:48.651 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.651 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.651 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.552772+0000", 2024-04-26T01:08:48.651 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0017207769999999999 2024-04-26T01:08:48.652 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.652 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.652 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.652 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.652 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.652 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 33/30 ECSubWriteReply(tid=14, last_complete=33'9, committed=1, applied=1))", 2024-04-26T01:08:48.652 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.551733+0000", 2024-04-26T01:08:48.652 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 49.068758969999998, 2024-04-26T01:08:48.652 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.40949085299999999, 2024-04-26T01:08:48.652 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.652 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.652 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.652 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.652 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.653 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.653 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551733+0000", 2024-04-26T01:08:48.653 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.653 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.653 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.653 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.653 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551733+0000", 2024-04-26T01:08:48.653 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.653 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.653 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.654 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.654 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551734+0000", 2024-04-26T01:08:48.654 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7199999999999998e-07 2024-04-26T01:08:48.654 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.654 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.654 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.654 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551739+0000", 2024-04-26T01:08:48.654 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9520000000000002e-06 2024-04-26T01:08:48.654 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.654 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.654 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.654 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.960976+0000", 2024-04-26T01:08:48.654 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.40923691400000001 2024-04-26T01:08:48.655 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.655 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.655 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.655 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.960989+0000", 2024-04-26T01:08:48.655 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3536000000000001e-05 2024-04-26T01:08:48.655 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.655 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.655 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.655 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.961157+0000", 2024-04-26T01:08:48.655 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016794600000000001 2024-04-26T01:08:48.655 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.655 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.655 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.655 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.961224+0000", 2024-04-26T01:08:48.656 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6733e-05 2024-04-26T01:08:48.656 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.656 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.656 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.656 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.656 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.656 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:251 3.5s0 3:acf4d671:::smithi16428812-10:head [append 0~0] snapc 4=[4,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:48.656 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:03.428674+0000", 2024-04-26T01:08:48.656 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 45.191817532999998, 2024-04-26T01:08:48.656 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.48188827299999998, 2024-04-26T01:08:48.656 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.656 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.656 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:48.656 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.656 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.657 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.657 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 251 2024-04-26T01:08:48.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.657 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.657 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.657 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428674+0000", 2024-04-26T01:08:48.657 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.657 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.657 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428674+0000", 2024-04-26T01:08:48.657 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.658 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428676+0000", 2024-04-26T01:08:48.658 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6559999999999999e-06 2024-04-26T01:08:48.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.658 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428685+0000", 2024-04-26T01:08:48.658 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0869999999999996e-06 2024-04-26T01:08:48.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.658 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428707+0000", 2024-04-26T01:08:48.658 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1365999999999998e-05 2024-04-26T01:08:48.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.659 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.659 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.659 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428723+0000", 2024-04-26T01:08:48.659 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6563e-05 2024-04-26T01:08:48.659 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.659 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.659 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.659 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.434334+0000", 2024-04-26T01:08:48.659 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056104170000000004 2024-04-26T01:08:48.659 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.659 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.659 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.659 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.434677+0000", 2024-04-26T01:08:48.659 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000343937 2024-04-26T01:08:48.660 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.660 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.660 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.436169+0000", 2024-04-26T01:08:48.660 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014915939999999999 2024-04-26T01:08:48.660 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.660 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.660 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.438266+0000", 2024-04-26T01:08:48.660 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002096746 2024-04-26T01:08:48.660 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.660 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:48.660 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.910445+0000", 2024-04-26T01:08:48.660 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.47217926300000002 2024-04-26T01:08:48.661 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.661 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.661 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.661 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.910563+0000", 2024-04-26T01:08:48.661 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000117644 2024-04-26T01:08:48.661 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.661 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.661 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.661 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.661 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.661 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:416 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-26T01:08:48.661 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.443980+0000", 2024-04-26T01:08:48.661 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 38.176512142999997, 2024-04-26T01:08:48.661 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.018880322000000001, 2024-04-26T01:08:48.661 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.662 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.662 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:48.662 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.662 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.662 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.662 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 416 2024-04-26T01:08:48.662 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.662 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.662 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.662 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.662 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443980+0000", 2024-04-26T01:08:48.662 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.662 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.662 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.663 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.663 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443980+0000", 2024-04-26T01:08:48.663 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.663 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.663 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.663 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443981+0000", 2024-04-26T01:08:48.663 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7000000000000003e-07 2024-04-26T01:08:48.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.663 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.663 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.663 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443984+0000", 2024-04-26T01:08:48.663 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.36e-06 2024-04-26T01:08:48.664 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.664 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.664 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443994+0000", 2024-04-26T01:08:48.664 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0285999999999999e-05 2024-04-26T01:08:48.664 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.664 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.664 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.444002+0000", 2024-04-26T01:08:48.665 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4970000000000004e-06 2024-04-26T01:08:48.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.665 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.665 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.665 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.454900+0000", 2024-04-26T01:08:48.665 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010897784000000001 2024-04-26T01:08:48.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.665 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.665 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:48.665 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.455102+0000", 2024-04-26T01:08:48.665 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00020183000000000001 2024-04-26T01:08:48.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.665 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.665 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.665 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.460460+0000", 2024-04-26T01:08:48.666 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0053580140000000004 2024-04-26T01:08:48.666 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.666 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.666 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.666 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.460760+0000", 2024-04-26T01:08:48.666 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00030007900000000002 2024-04-26T01:08:48.666 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.666 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.666 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.666 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.462860+0000", 2024-04-26T01:08:48.666 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021005020000000002 2024-04-26T01:08:48.666 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.666 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.666 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.667 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.667 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.667 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.3s3 44/30 ECSubWrite(tid=198, reqid=client.4521.0:420, at_version=44'13, trim_to=0'0, roll_forward_to=44'11))", 2024-04-26T01:08:48.667 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.466337+0000", 2024-04-26T01:08:48.667 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 38.154155385999999, 2024-04-26T01:08:48.667 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.57864738100000002, 2024-04-26T01:08:48.667 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.667 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.667 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.667 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.667 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.667 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.667 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466337+0000", 2024-04-26T01:08:48.667 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.668 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.668 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.668 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.668 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466337+0000", 2024-04-26T01:08:48.668 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.668 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.668 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.668 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.668 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466338+0000", 2024-04-26T01:08:48.668 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4479999999999999e-06 2024-04-26T01:08:48.668 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.668 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.668 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.668 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466374+0000", 2024-04-26T01:08:48.668 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6137999999999999e-05 2024-04-26T01:08:48.669 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.669 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.669 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.669 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.043926+0000", 2024-04-26T01:08:48.669 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.57755188800000001 2024-04-26T01:08:48.669 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.669 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.669 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.669 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.043944+0000", 2024-04-26T01:08:48.669 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7745999999999999e-05 2024-04-26T01:08:48.669 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.669 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.669 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.669 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044049+0000", 2024-04-26T01:08:48.669 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010500700000000001 2024-04-26T01:08:48.670 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.670 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.670 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.670 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044068+0000", 2024-04-26T01:08:48.670 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8984999999999999e-05 2024-04-26T01:08:48.670 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.671 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.671 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044949+0000", 2024-04-26T01:08:48.671 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00088138400000000003 2024-04-26T01:08:48.671 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.671 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.671 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044984+0000", 2024-04-26T01:08:48.671 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4785e-05 2024-04-26T01:08:48.671 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.671 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.671 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.671 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.672 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:554 3.5s0 3:a661dd24:::smithi16428812-43:head [copy-from ver 26] snapc e=[e,d,b,7] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-04-26T01:08:48.672 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:21.094110+0000", 2024-04-26T01:08:48.672 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 27.526381597, 2024-04-26T01:08:48.672 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.050017671, 2024-04-26T01:08:48.672 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.672 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.672 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:48.672 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.672 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.672 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.672 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 554 2024-04-26T01:08:48.672 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.672 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.672 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.672 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.673 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094110+0000", 2024-04-26T01:08:48.673 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.673 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.673 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.673 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.673 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094110+0000", 2024-04-26T01:08:48.673 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.673 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.673 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.673 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.673 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094114+0000", 2024-04-26T01:08:48.673 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.89e-06 2024-04-26T01:08:48.673 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.673 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.673 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.674 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094134+0000", 2024-04-26T01:08:48.674 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0007e-05 2024-04-26T01:08:48.674 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.674 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.674 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.674 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094181+0000", 2024-04-26T01:08:48.674 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6731999999999999e-05 2024-04-26T01:08:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094205+0000", 2024-04-26T01:08:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4326999999999999e-05 2024-04-26T01:08:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094455+0000", 2024-04-26T01:08:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024929999999999999 2024-04-26T01:08:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2024-04-26T01:08:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094477+0000", 2024-04-26T01:08:48.676 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2646e-05 2024-04-26T01:08:48.676 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.676 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.676 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.676 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105141+0000", 2024-04-26T01:08:48.676 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010664167 2024-04-26T01:08:48.676 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.676 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.676 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.676 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105392+0000", 2024-04-26T01:08:48.676 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00025070700000000002 2024-04-26T01:08:48.676 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.676 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.676 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.676 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.135272+0000", 2024-04-26T01:08:48.677 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.029880263000000001 2024-04-26T01:08:48.677 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.677 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.677 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.677 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.142569+0000", 2024-04-26T01:08:48.677 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0072964270000000003 2024-04-26T01:08:48.677 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.677 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.677 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:48.677 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.142755+0000", 2024-04-26T01:08:48.677 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00018603399999999999 2024-04-26T01:08:48.677 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.677 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.677 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.677 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.144128+0000", 2024-04-26T01:08:48.678 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0013731710000000001 2024-04-26T01:08:48.678 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.678 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.678 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.678 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.678 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.678 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:555 3.5s0 3:a661dd24:::smithi16428812-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e55)", 2024-04-26T01:08:48.678 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:21.094258+0000", 2024-04-26T01:08:48.678 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 27.526234431999999, 2024-04-26T01:08:48.678 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.045831917999999999, 2024-04-26T01:08:48.678 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.678 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.679 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:48.679 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.679 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.679 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.679 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 555 2024-04-26T01:08:48.679 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.679 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.679 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.679 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.679 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094258+0000", 2024-04-26T01:08:48.679 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.679 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.679 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.679 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.679 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094258+0000", 2024-04-26T01:08:48.680 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.680 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.680 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.680 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.680 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094259+0000", 2024-04-26T01:08:48.680 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7370000000000001e-06 2024-04-26T01:08:48.680 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.680 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.680 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.681 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094268+0000", 2024-04-26T01:08:48.681 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8359999999999998e-06 2024-04-26T01:08:48.681 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.681 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.681 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.681 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094289+0000", 2024-04-26T01:08:48.681 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0937e-05 2024-04-26T01:08:48.681 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.681 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.681 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.681 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094299+0000", 2024-04-26T01:08:48.681 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.029e-05 2024-04-26T01:08:48.681 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.682 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.682 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.682 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094609+0000", 2024-04-26T01:08:48.682 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00031005900000000001 2024-04-26T01:08:48.682 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.682 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.682 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:48.682 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094619+0000", 2024-04-26T01:08:48.682 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9799999999999993e-06 2024-04-26T01:08:48.682 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.682 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.682 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.683 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105868+0000", 2024-04-26T01:08:48.683 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011248555 2024-04-26T01:08:48.683 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.683 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.683 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2024-04-26T01:08:48.683 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105946+0000", 2024-04-26T01:08:48.683 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8005999999999998e-05 2024-04-26T01:08:48.683 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.683 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.683 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.683 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.139672+0000", 2024-04-26T01:08:48.683 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.033725805999999997 2024-04-26T01:08:48.683 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.683 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.684 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.684 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.139869+0000", 2024-04-26T01:08:48.684 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019683 2024-04-26T01:08:48.684 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.684 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.684 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.684 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.140089+0000", 2024-04-26T01:08:48.684 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022088199999999999 2024-04-26T01:08:48.684 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.684 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.684 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.685 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.685 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.685 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s4 71/44 ECSubWrite(tid=234, reqid=client.4521.0:741, at_version=71'32, trim_to=0'0, roll_forward_to=71'28))", 2024-04-26T01:08:48.685 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:37.153424+0000", 2024-04-26T01:08:48.685 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 11.467067528999999, 2024-04-26T01:08:48.685 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.162277859, 2024-04-26T01:08:48.685 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.685 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.685 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.685 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.685 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.685 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.685 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.153424+0000", 2024-04-26T01:08:48.686 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.686 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.686 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.686 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.686 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.153424+0000", 2024-04-26T01:08:48.686 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.686 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.686 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.686 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.686 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.153426+0000", 2024-04-26T01:08:48.686 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8729999999999999e-06 2024-04-26T01:08:48.686 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.686 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.686 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.687 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.153470+0000", 2024-04-26T01:08:48.687 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3185000000000001e-05 2024-04-26T01:08:48.687 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.687 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.687 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.313873+0000", 2024-04-26T01:08:48.687 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.160403237 2024-04-26T01:08:48.687 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.687 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.687 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.313888+0000", 2024-04-26T01:08:48.687 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4796e-05 2024-04-26T01:08:48.687 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.687 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.688 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.314082+0000", 2024-04-26T01:08:48.688 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019490800000000001 2024-04-26T01:08:48.688 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.688 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.688 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.688 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.314120+0000", 2024-04-26T01:08:48.688 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7954999999999998e-05 2024-04-26T01:08:48.688 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.688 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.688 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.688 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.315656+0000", 2024-04-26T01:08:48.688 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0015357960000000001 2024-04-26T01:08:48.688 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.688 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.688 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.689 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.315702+0000", 2024-04-26T01:08:48.689 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6109000000000002e-05 2024-04-26T01:08:48.689 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.689 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.689 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.689 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.689 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.689 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.0s2 71/30 ECSubWrite(tid=78, reqid=client.4521.0:774, at_version=71'64, trim_to=0'0, roll_forward_to=71'61))", 2024-04-26T01:08:48.689 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:37.284626+0000", 2024-04-26T01:08:48.689 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 11.335865817, 2024-04-26T01:08:48.689 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.30429075900000002, 2024-04-26T01:08:48.689 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.689 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.689 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.690 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.690 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.690 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.690 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.284626+0000", 2024-04-26T01:08:48.690 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.690 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.690 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.690 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.690 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.284626+0000", 2024-04-26T01:08:48.690 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.690 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.690 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.690 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.690 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.284628+0000", 2024-04-26T01:08:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5570000000000001e-06 2024-04-26T01:08:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.284659+0000", 2024-04-26T01:08:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1751999999999997e-05 2024-04-26T01:08:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.587683+0000", 2024-04-26T01:08:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.30302400200000001 2024-04-26T01:08:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.587698+0000", 2024-04-26T01:08:48.692 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5006e-05 2024-04-26T01:08:48.692 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.692 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.692 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.692 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.587817+0000", 2024-04-26T01:08:48.692 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011894499999999999 2024-04-26T01:08:48.692 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.692 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.692 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.692 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.587840+0000", 2024-04-26T01:08:48.692 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2218999999999999e-05 2024-04-26T01:08:48.692 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.692 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.692 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.692 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.588819+0000", 2024-04-26T01:08:48.693 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00097946199999999995 2024-04-26T01:08:48.693 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.693 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.693 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.693 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.588917+0000", 2024-04-26T01:08:48.693 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7816e-05 2024-04-26T01:08:48.693 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.693 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.693 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.693 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.693 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.693 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s4 71/44 ECSubWrite(tid=238, reqid=client.4521.0:741, at_version=0'0, trim_to=0'0, roll_forward_to=71'32))", 2024-04-26T01:08:48.693 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:37.316333+0000", 2024-04-26T01:08:48.693 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 11.304159030999999, 2024-04-26T01:08:48.694 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.27762489099999998, 2024-04-26T01:08:48.694 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.694 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.694 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:48.694 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.694 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.694 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.694 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.316333+0000", 2024-04-26T01:08:48.694 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.694 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.694 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.694 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.694 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.316333+0000", 2024-04-26T01:08:48.694 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.695 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.695 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.695 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.695 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.316334+0000", 2024-04-26T01:08:48.695 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4169999999999999e-06 2024-04-26T01:08:48.695 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.695 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.695 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.695 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.316361+0000", 2024-04-26T01:08:48.696 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6299e-05 2024-04-26T01:08:48.696 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.696 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.696 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.696 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.592405+0000", 2024-04-26T01:08:48.696 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.27604479100000001 2024-04-26T01:08:48.696 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.696 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.697 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.697 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.592428+0000", 2024-04-26T01:08:48.697 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2039e-05 2024-04-26T01:08:48.697 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.697 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.697 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.697 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.592591+0000", 2024-04-26T01:08:48.697 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016372700000000001 2024-04-26T01:08:48.697 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.697 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.697 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.697 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.592613+0000", 2024-04-26T01:08:48.697 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2118999999999999e-05 2024-04-26T01:08:48.697 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.698 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.698 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.698 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.593876+0000", 2024-04-26T01:08:48.698 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012630989999999999 2024-04-26T01:08:48.698 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.698 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.698 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.698 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.593958+0000", 2024-04-26T01:08:48.698 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.14e-05 2024-04-26T01:08:48.698 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.698 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.698 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.698 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.699 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.699 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:785 3.5s0 3:a8ed6f44:::smithi16428812-28:head [copy-from ver 62] snapc 15=[15] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:08:48.699 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:38.277776+0000", 2024-04-26T01:08:48.699 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 10.342716326, 2024-04-26T01:08:48.699 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.036482200999999999, 2024-04-26T01:08:48.699 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.699 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.699 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:48.699 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.699 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.699 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.699 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 785 2024-04-26T01:08:48.699 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.699 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.699 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.700 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.700 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277776+0000", 2024-04-26T01:08:48.700 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.700 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.700 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.700 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.700 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277776+0000", 2024-04-26T01:08:48.700 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.700 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.700 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.700 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.700 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277778+0000", 2024-04-26T01:08:48.700 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.866e-06 2024-04-26T01:08:48.700 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.701 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.701 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.701 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277785+0000", 2024-04-26T01:08:48.701 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2629999999999997e-06 2024-04-26T01:08:48.701 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.701 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.701 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.701 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277800+0000", 2024-04-26T01:08:48.701 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4888999999999999e-05 2024-04-26T01:08:48.701 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.701 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.701 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.701 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277807+0000", 2024-04-26T01:08:48.701 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8170000000000001e-06 2024-04-26T01:08:48.702 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.702 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.702 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.702 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277869+0000", 2024-04-26T01:08:48.702 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2844999999999999e-05 2024-04-26T01:08:48.702 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.702 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.702 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.702 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277990+0000", 2024-04-26T01:08:48.702 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012097899999999999 2024-04-26T01:08:48.702 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.702 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.702 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:48.702 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.302496+0000", 2024-04-26T01:08:48.702 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.024505859000000001 2024-04-26T01:08:48.703 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.703 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.703 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:48.703 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.311498+0000", 2024-04-26T01:08:48.703 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0090014369999999993 2024-04-26T01:08:48.703 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.703 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.703 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:48.703 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.314125+0000", 2024-04-26T01:08:48.703 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0026272930000000002 2024-04-26T01:08:48.703 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.703 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.703 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.703 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.314258+0000", 2024-04-26T01:08:48.704 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000132953 2024-04-26T01:08:48.704 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.704 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.704 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.704 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.704 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.704 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:786 3.5s0 3:a8ed6f44:::smithi16428812-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:08:48.704 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:38.277827+0000", 2024-04-26T01:08:48.704 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 10.342664902999999, 2024-04-26T01:08:48.704 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.033385495000000001, 2024-04-26T01:08:48.704 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.704 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.704 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:48.704 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.705 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.705 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.705 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 786 2024-04-26T01:08:48.705 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.705 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.705 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.705 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.705 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277827+0000", 2024-04-26T01:08:48.705 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.705 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.706 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.706 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.706 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277827+0000", 2024-04-26T01:08:48.706 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.706 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.706 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.706 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.706 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277828+0000", 2024-04-26T01:08:48.706 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6199999999999997e-07 2024-04-26T01:08:48.706 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.706 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.706 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.706 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277832+0000", 2024-04-26T01:08:48.706 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1400000000000002e-06 2024-04-26T01:08:48.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.707 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.707 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277841+0000", 2024-04-26T01:08:48.707 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0699999999999996e-06 2024-04-26T01:08:48.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.707 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.707 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277848+0000", 2024-04-26T01:08:48.707 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1119999999999998e-06 2024-04-26T01:08:48.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.707 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.707 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.278246+0000", 2024-04-26T01:08:48.708 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00039790799999999999 2024-04-26T01:08:48.708 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.708 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.708 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2024-04-26T01:08:48.708 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.278408+0000", 2024-04-26T01:08:48.708 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016165099999999999 2024-04-26T01:08:48.708 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.708 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.708 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.708 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.310714+0000", 2024-04-26T01:08:48.708 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.032306058999999998 2024-04-26T01:08:48.708 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.708 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.708 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.709 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.311023+0000", 2024-04-26T01:08:48.709 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00030885 2024-04-26T01:08:48.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.709 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.709 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.311213+0000", 2024-04-26T01:08:48.709 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00018994300000000001 2024-04-26T01:08:48.709 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.709 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.709 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.709 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:863 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1753088,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-04-26T01:08:48.709 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:43.278882+0000", 2024-04-26T01:08:48.710 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 5.3416096020000001, 2024-04-26T01:08:48.710 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.018957168, 2024-04-26T01:08:48.710 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.710 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.710 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:48.710 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.710 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.710 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.710 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 863 2024-04-26T01:08:48.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.710 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.710 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.710 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.710 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278882+0000", 2024-04-26T01:08:48.711 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.711 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.711 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.711 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.711 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278882+0000", 2024-04-26T01:08:48.711 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.711 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.711 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.711 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.711 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278886+0000", 2024-04-26T01:08:48.711 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8829999999999999e-06 2024-04-26T01:08:48.711 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.711 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.711 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.712 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278905+0000", 2024-04-26T01:08:48.712 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9168999999999999e-05 2024-04-26T01:08:48.712 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.712 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.712 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.712 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278950+0000", 2024-04-26T01:08:48.712 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5028000000000003e-05 2024-04-26T01:08:48.712 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.712 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.712 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.712 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278974+0000", 2024-04-26T01:08:48.712 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3682000000000001e-05 2024-04-26T01:08:48.712 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.713 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.713 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.713 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279142+0000", 2024-04-26T01:08:48.713 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016799499999999999 2024-04-26T01:08:48.713 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.713 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.713 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2024-04-26T01:08:48.713 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279164+0000", 2024-04-26T01:08:48.713 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2039e-05 2024-04-26T01:08:48.713 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.713 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.713 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.713 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.283284+0000", 2024-04-26T01:08:48.714 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0041193239999999997 2024-04-26T01:08:48.714 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.714 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.714 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.714 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.283451+0000", 2024-04-26T01:08:48.714 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016756799999999999 2024-04-26T01:08:48.714 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.714 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.714 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.715 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.297840+0000", 2024-04-26T01:08:48.715 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.01438848 2024-04-26T01:08:48.715 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.715 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.715 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.715 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.715 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.715 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:864 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1753088,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-04-26T01:08:48.715 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:43.279019+0000", 2024-04-26T01:08:48.715 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 5.3414726620000001, 2024-04-26T01:08:48.715 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.021489656999999999, 2024-04-26T01:08:48.715 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.715 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.715 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:48.716 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.716 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.716 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.716 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 864 2024-04-26T01:08:48.716 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.716 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.716 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.716 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.716 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279019+0000", 2024-04-26T01:08:48.716 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.716 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.716 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.717 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.717 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279019+0000", 2024-04-26T01:08:48.717 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.717 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.717 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.717 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279021+0000", 2024-04-26T01:08:48.717 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.243e-06 2024-04-26T01:08:48.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.717 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.718 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.718 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279028+0000", 2024-04-26T01:08:48.718 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0899999999999999e-06 2024-04-26T01:08:48.718 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.718 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.718 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279044+0000", 2024-04-26T01:08:48.718 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6545000000000002e-05 2024-04-26T01:08:48.718 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.718 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.719 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279053+0000", 2024-04-26T01:08:48.719 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1600000000000004e-06 2024-04-26T01:08:48.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.719 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.719 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.719 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279342+0000", 2024-04-26T01:08:48.719 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028902300000000001 2024-04-26T01:08:48.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.719 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.719 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:48.719 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279354+0000", 2024-04-26T01:08:48.719 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1542999999999999e-05 2024-04-26T01:08:48.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.719 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.720 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.720 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.284845+0000", 2024-04-26T01:08:48.720 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0054915240000000002 2024-04-26T01:08:48.720 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.720 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.720 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.720 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.284995+0000", 2024-04-26T01:08:48.720 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014962299999999999 2024-04-26T01:08:48.720 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.720 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.720 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.720 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.300509+0000", 2024-04-26T01:08:48.720 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015513905999999999 2024-04-26T01:08:48.720 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.720 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.721 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.721 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.721 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.721 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:865 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1753088,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-04-26T01:08:48.721 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:43.279067+0000", 2024-04-26T01:08:48.721 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 5.3414249050000002, 2024-04-26T01:08:48.721 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.023511127999999999, 2024-04-26T01:08:48.721 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:48.721 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:48.721 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:48.721 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:48.721 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:48.721 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:48.722 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 865 2024-04-26T01:08:48.722 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.722 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:48.722 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.722 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:48.722 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279067+0000", 2024-04-26T01:08:48.722 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.722 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.722 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.722 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:48.722 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279067+0000", 2024-04-26T01:08:48.722 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:48.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.723 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:48.723 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279068+0000", 2024-04-26T01:08:48.723 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9699999999999995e-07 2024-04-26T01:08:48.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.723 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:48.723 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279072+0000", 2024-04-26T01:08:48.723 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2400000000000001e-06 2024-04-26T01:08:48.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.723 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:48.724 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279082+0000", 2024-04-26T01:08:48.724 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6430000000000007e-06 2024-04-26T01:08:48.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.724 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.724 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:48.724 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279089+0000", 2024-04-26T01:08:48.724 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0690000000000004e-06 2024-04-26T01:08:48.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.724 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.724 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.724 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279431+0000", 2024-04-26T01:08:48.724 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034206800000000001 2024-04-26T01:08:48.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.725 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:48.725 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279454+0000", 2024-04-26T01:08:48.725 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3448999999999999e-05 2024-04-26T01:08:48.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.725 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:48.725 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.287223+0000", 2024-04-26T01:08:48.725 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077688849999999997 2024-04-26T01:08:48.726 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.726 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.726 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:48.726 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.287470+0000", 2024-04-26T01:08:48.726 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024656199999999997 2024-04-26T01:08:48.727 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:48.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:48.727 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:48.727 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.302578+0000", 2024-04-26T01:08:48.727 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015108415 2024-04-26T01:08:48.727 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.727 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.727 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.727 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:48.727 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:48.728 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:48.729 INFO:teuthology.orchestra.run.smithi164.stdout:3.ds1 2024-04-26T01:08:48.729 INFO:teuthology.orchestra.run.smithi164.stdout:3.as0 2024-04-26T01:08:48.729 INFO:teuthology.orchestra.run.smithi164.stdout:3.9s3 2024-04-26T01:08:48.729 INFO:teuthology.orchestra.run.smithi164.stdout:3.3s6 2024-04-26T01:08:48.729 INFO:teuthology.orchestra.run.smithi164.stdout:3.2s2 2024-04-26T01:08:48.729 INFO:teuthology.orchestra.run.smithi164.stdout:2.2 2024-04-26T01:08:48.729 INFO:teuthology.orchestra.run.smithi164.stdout:2.0 2024-04-26T01:08:48.730 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:08:48.731 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:48.724+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:48.830 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:48.827+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:48.856 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:48.856 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:48.857 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:08:48.857 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:48.873 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:08:48.931 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:48.931+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:48.999 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:48.999 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:48.999 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2024-04-26T01:08:48.999 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2024-04-26T01:08:48.999 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:49.013 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:08:49.033 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:49.033+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:49.133 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:49.132+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:49.181 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 3.3s6 --file /var/log/ceph/exp.3.3s6.3 2024-04-26T01:08:49.181 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:49.181 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2024-04-26T01:08:49.181 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2024-04-26T01:08:49.182 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [] 2024-04-26T01:08:49.182 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:49.196 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:08:49.234 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:49.229+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:49.328 INFO:tasks.rados.rados.0.smithi164.stdout:363: done (6 left) 2024-04-26T01:08:49.329 INFO:tasks.rados.rados.0.smithi164.stdout:364: snap_create 2024-04-26T01:08:49.329 INFO:tasks.rados.rados.0.smithi164.stdout:360: finishing write tid 1 to smithi16428812-45 2024-04-26T01:08:49.329 INFO:tasks.rados.rados.0.smithi164.stdout:362: finishing rollback tid 0 to smithi16428812-33 2024-04-26T01:08:49.329 INFO:tasks.rados.rados.0.smithi164.stdout:362: finishing rollback tid 1 to smithi16428812-33 2024-04-26T01:08:49.329 INFO:tasks.rados.rados.0.smithi164.stdout:362: finishing rollback tid 2 to smithi16428812-33 2024-04-26T01:08:49.329 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 33 v 98 (ObjNum 147 snap 23 seq_num 147) dirty exists 2024-04-26T01:08:49.329 INFO:tasks.rados.rados.0.smithi164.stdout:360: finishing write tid 2 to smithi16428812-45 2024-04-26T01:08:49.329 INFO:tasks.rados.rados.0.smithi164.stdout:361: finishing copy_from racing read to smithi16428812-40 2024-04-26T01:08:49.329 INFO:tasks.rados.rados.0.smithi164.stdout:344: expect (ObjNum 69 snap 4 seq_num 69) 2024-04-26T01:08:49.335 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:49.335+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:49.340 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:49.340 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:49.340 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-04-26T01:08:49.340 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:49.362 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:08:49.417 INFO:tasks.rados.rados.0.smithi164.stdout:361: finishing copy_from to smithi16428812-40 2024-04-26T01:08:49.418 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 40 v 15 (ObjNum 71 snap 4 seq_num 71) dirty exists 2024-04-26T01:08:49.418 INFO:tasks.rados.rados.0.smithi164.stdout:360: finishing write tid 3 to smithi16428812-45 2024-04-26T01:08:49.418 INFO:tasks.rados.rados.0.smithi164.stdout:360: oid 45 updating version 0 to 110 2024-04-26T01:08:49.418 INFO:tasks.rados.rados.0.smithi164.stdout:360: oid 45 updating version 110 to 111 2024-04-26T01:08:49.418 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 45 v 111 (ObjNum 150 snap 27 seq_num 150) dirty exists 2024-04-26T01:08:49.418 INFO:tasks.rados.rados.0.smithi164.stdout:360: left oid 45 (ObjNum 150 snap 27 seq_num 150) 2024-04-26T01:08:49.418 INFO:tasks.rados.rados.0.smithi164.stdout:354: finishing copy_from racing read to smithi16428812-8 2024-04-26T01:08:49.418 INFO:tasks.rados.rados.0.smithi164.stdout:354: finishing copy_from to smithi16428812-8 2024-04-26T01:08:49.418 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 8 v 32 (ObjNum 127 snap 19 seq_num 127) dirty exists 2024-04-26T01:08:49.418 INFO:tasks.rados.rados.0.smithi164.stdout:346: finishing copy_from racing read to smithi16428812-37 2024-04-26T01:08:49.418 INFO:tasks.rados.rados.0.smithi164.stdout:346: finishing copy_from to smithi16428812-37 2024-04-26T01:08:49.418 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 37 v 22 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-26T01:08:49.436 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:49.432+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:49.481 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:49.481 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:49.481 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-04-26T01:08:49.481 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-04-26T01:08:49.481 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:49.495 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:08:49.537 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:49.535+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:49.638 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:49.635+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:49.661 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:49.661 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:49.661 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:49.661 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:49.661 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.662 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.7s3 33/30 ECSubWrite(tid=9, reqid=client.4521.0:38, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:49.662 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.325556+0000", 2024-04-26T01:08:49.662 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 50.332520414999998, 2024-04-26T01:08:49.662 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.028941706000000001, 2024-04-26T01:08:49.662 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.662 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.662 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:49.662 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.663 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.663 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.663 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325556+0000", 2024-04-26T01:08:49.663 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.663 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.663 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.663 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.663 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325556+0000", 2024-04-26T01:08:49.665 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.665 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.665 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.665 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.665 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325557+0000", 2024-04-26T01:08:49.666 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.221e-06 2024-04-26T01:08:49.666 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.666 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.666 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.666 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.325581+0000", 2024-04-26T01:08:49.666 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3502e-05 2024-04-26T01:08:49.667 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.667 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.667 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.667 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353123+0000", 2024-04-26T01:08:49.667 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.027542540000000001 2024-04-26T01:08:49.667 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.667 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.667 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.667 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353131+0000", 2024-04-26T01:08:49.667 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6909999999999997e-06 2024-04-26T01:08:49.667 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.667 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.667 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.667 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353823+0000", 2024-04-26T01:08:49.667 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00069229100000000004 2024-04-26T01:08:49.668 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.668 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.668 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:49.668 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.353835+0000", 2024-04-26T01:08:49.668 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1507e-05 2024-04-26T01:08:49.668 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.668 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.668 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:49.668 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.354460+0000", 2024-04-26T01:08:49.668 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00062488599999999995 2024-04-26T01:08:49.668 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.668 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.668 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.668 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.354498+0000", 2024-04-26T01:08:49.669 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8068000000000002e-05 2024-04-26T01:08:49.669 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.669 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.669 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.669 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.669 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.669 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.7s3 33/30 ECSubWrite(tid=16, reqid=client.4521.0:65, at_version=33'8, trim_to=0'0, roll_forward_to=33'6))", 2024-04-26T01:08:49.669 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.366487+0000", 2024-04-26T01:08:49.669 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 50.291589088999999, 2024-04-26T01:08:49.669 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.079262088999999994, 2024-04-26T01:08:49.669 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.669 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.669 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:49.669 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.669 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.670 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.670 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366487+0000", 2024-04-26T01:08:49.670 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.670 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.670 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.670 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.670 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366487+0000", 2024-04-26T01:08:49.670 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.670 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.670 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.670 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.670 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366488+0000", 2024-04-26T01:08:49.670 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6700000000000002e-07 2024-04-26T01:08:49.670 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.671 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.671 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.671 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.366510+0000", 2024-04-26T01:08:49.671 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1591999999999999e-05 2024-04-26T01:08:49.671 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.671 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.671 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.671 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444763+0000", 2024-04-26T01:08:49.671 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.078252639999999998 2024-04-26T01:08:49.671 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.671 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.671 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.671 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444773+0000", 2024-04-26T01:08:49.671 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0134e-05 2024-04-26T01:08:49.671 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444870+0000", 2024-04-26T01:08:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6847000000000003e-05 2024-04-26T01:08:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.444880+0000", 2024-04-26T01:08:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0312999999999999e-05 2024-04-26T01:08:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.445715+0000", 2024-04-26T01:08:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00083498700000000003 2024-04-26T01:08:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.673 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.673 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.673 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.445750+0000", 2024-04-26T01:08:49.673 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4609e-05 2024-04-26T01:08:49.673 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.673 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.673 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.673 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.673 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.673 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpRead(3.cs2 33/30 ECSubRead(tid=19, to_read={3:3ca9ce7a:::smithi16428812-23:head=0,4096,0}, subchunks={3:3ca9ce7a:::smithi16428812-23:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:49.673 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.374332+0000", 2024-04-26T01:08:49.673 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 50.283744466999998, 2024-04-26T01:08:49.673 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.023990841999999998, 2024-04-26T01:08:49.673 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.674 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.674 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:49.674 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.674 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.674 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.674 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374332+0000", 2024-04-26T01:08:49.674 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.674 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.674 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.674 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.674 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374332+0000", 2024-04-26T01:08:49.674 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.674 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.674 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.675 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.675 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374333+0000", 2024-04-26T01:08:49.675 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0070000000000001e-06 2024-04-26T01:08:49.675 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.675 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.675 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.675 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.374345+0000", 2024-04-26T01:08:49.675 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1853e-05 2024-04-26T01:08:49.675 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.675 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.675 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.675 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398055+0000", 2024-04-26T01:08:49.675 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.023710078999999998 2024-04-26T01:08:49.675 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.676 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.676 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.676 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398067+0000", 2024-04-26T01:08:49.676 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2455e-05 2024-04-26T01:08:49.676 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.676 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.676 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.676 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398160+0000", 2024-04-26T01:08:49.676 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2682000000000005e-05 2024-04-26T01:08:49.676 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.676 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.676 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.676 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.398323+0000", 2024-04-26T01:08:49.676 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016276600000000001 2024-04-26T01:08:49.677 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.677 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.677 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.677 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.677 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.677 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:240 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:49.677 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.342427+0000", 2024-04-26T01:08:49.677 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 46.315649577999999, 2024-04-26T01:08:49.677 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.019852153000000001, 2024-04-26T01:08:49.677 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.677 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.677 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:49.677 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:49.677 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:49.677 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:49.678 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 240 2024-04-26T01:08:49.678 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.678 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.678 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.678 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.678 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342427+0000", 2024-04-26T01:08:49.678 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.678 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.678 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.678 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.678 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342427+0000", 2024-04-26T01:08:49.678 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.678 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.679 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.679 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.679 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342428+0000", 2024-04-26T01:08:49.679 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2200000000000003e-07 2024-04-26T01:08:49.679 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.679 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.679 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.679 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342435+0000", 2024-04-26T01:08:49.679 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0180000000000004e-06 2024-04-26T01:08:49.679 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.679 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.679 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.679 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342451+0000", 2024-04-26T01:08:49.679 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6011e-05 2024-04-26T01:08:49.679 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.680 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.680 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.680 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342459+0000", 2024-04-26T01:08:49.680 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1370000000000002e-06 2024-04-26T01:08:49.680 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.680 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.680 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.680 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342766+0000", 2024-04-26T01:08:49.680 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00030684000000000001 2024-04-26T01:08:49.680 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.680 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.680 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:49.680 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342784+0000", 2024-04-26T01:08:49.680 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7773000000000001e-05 2024-04-26T01:08:49.681 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.681 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.681 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.681 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.349542+0000", 2024-04-26T01:08:49.681 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0067586349999999998 2024-04-26T01:08:49.681 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.681 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.681 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:49.681 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.349627+0000", 2024-04-26T01:08:49.681 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4790999999999995e-05 2024-04-26T01:08:49.681 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.681 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.681 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.681 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.362279+0000", 2024-04-26T01:08:49.681 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012652126 2024-04-26T01:08:49.682 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.682 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.682 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.682 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.682 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.682 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:241 3.6s0 3:6387e095:::smithi16428812-24:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:49.682 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.342661+0000", 2024-04-26T01:08:49.682 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 46.315415727999998, 2024-04-26T01:08:49.682 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.020164171000000002, 2024-04-26T01:08:49.682 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.682 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.682 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:49.682 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:49.683 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:49.683 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:49.683 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 241 2024-04-26T01:08:49.683 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.683 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.683 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.683 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.683 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342661+0000", 2024-04-26T01:08:49.683 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.683 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.683 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.683 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.683 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342661+0000", 2024-04-26T01:08:49.683 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.683 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.684 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.684 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.684 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342662+0000", 2024-04-26T01:08:49.684 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0100000000000003e-07 2024-04-26T01:08:49.684 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.684 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.684 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.684 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342668+0000", 2024-04-26T01:08:49.684 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2369999999999997e-06 2024-04-26T01:08:49.684 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.684 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.684 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.684 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342682+0000", 2024-04-26T01:08:49.684 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4166e-05 2024-04-26T01:08:49.685 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.685 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.685 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.685 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342691+0000", 2024-04-26T01:08:49.685 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8000000000000004e-06 2024-04-26T01:08:49.685 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.685 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.685 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.685 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342844+0000", 2024-04-26T01:08:49.685 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015316599999999999 2024-04-26T01:08:49.685 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.685 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.685 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:49.685 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.342853+0000", 2024-04-26T01:08:49.685 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9700000000000005e-06 2024-04-26T01:08:49.686 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.686 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.686 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.686 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.350630+0000", 2024-04-26T01:08:49.686 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0077770890000000001 2024-04-26T01:08:49.686 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.686 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.686 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:49.686 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.350718+0000", 2024-04-26T01:08:49.686 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8156000000000001e-05 2024-04-26T01:08:49.686 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.686 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.686 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.686 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.362825+0000", 2024-04-26T01:08:49.687 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012106686 2024-04-26T01:08:49.687 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.687 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.687 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.687 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.687 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.687 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs2 41/30 ECSubWrite(tid=91, reqid=client.4521.0:345, at_version=41'27, trim_to=0'0, roll_forward_to=41'26))", 2024-04-26T01:08:49.687 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:07.610935+0000", 2024-04-26T01:08:49.687 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 42.047141205000003, 2024-04-26T01:08:49.687 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.629675231, 2024-04-26T01:08:49.687 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.687 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.687 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:49.687 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.687 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.688 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.688 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.610935+0000", 2024-04-26T01:08:49.688 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.688 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.688 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.688 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.688 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.610935+0000", 2024-04-26T01:08:49.688 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.688 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.688 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.688 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.688 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.610938+0000", 2024-04-26T01:08:49.688 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.39e-06 2024-04-26T01:08:49.688 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.688 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.689 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.689 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:07.611405+0000", 2024-04-26T01:08:49.689 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00046744899999999999 2024-04-26T01:08:49.689 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.689 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.689 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.689 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238064+0000", 2024-04-26T01:08:49.689 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.62665866100000001 2024-04-26T01:08:49.689 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.689 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.689 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.689 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238080+0000", 2024-04-26T01:08:49.689 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6623e-05 2024-04-26T01:08:49.689 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.690 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.690 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.690 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238214+0000", 2024-04-26T01:08:49.690 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013316900000000001 2024-04-26T01:08:49.690 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.690 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.690 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:49.690 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.238235+0000", 2024-04-26T01:08:49.690 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0937e-05 2024-04-26T01:08:49.690 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.690 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.690 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:49.690 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.240510+0000", 2024-04-26T01:08:49.690 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022752860000000001 2024-04-26T01:08:49.690 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.691 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.691 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.691 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.240611+0000", 2024-04-26T01:08:49.691 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000100716 2024-04-26T01:08:49.691 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.691 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.691 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.691 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.691 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.691 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 43/30 ECSubWriteReply(tid=33, last_complete=43'17, committed=1, applied=1))", 2024-04-26T01:08:49.691 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.473084+0000", 2024-04-26T01:08:49.691 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 40.184992962000003, 2024-04-26T01:08:49.691 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.913281966, 2024-04-26T01:08:49.691 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.692 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.692 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:49.692 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.692 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.692 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.692 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.473084+0000", 2024-04-26T01:08:49.692 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.692 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.692 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.692 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.692 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.473084+0000", 2024-04-26T01:08:49.692 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.692 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.692 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.693 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.693 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.473084+0000", 2024-04-26T01:08:49.693 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6300000000000005e-07 2024-04-26T01:08:49.693 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.693 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.693 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.693 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.473088+0000", 2024-04-26T01:08:49.693 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7620000000000001e-06 2024-04-26T01:08:49.693 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.693 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.693 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.693 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.386034+0000", 2024-04-26T01:08:49.693 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.91294643399999997 2024-04-26T01:08:49.693 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.693 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.694 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.694 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.386055+0000", 2024-04-26T01:08:49.694 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0492999999999998e-05 2024-04-26T01:08:49.694 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.694 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.694 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.694 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.386234+0000", 2024-04-26T01:08:49.694 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000179259 2024-04-26T01:08:49.694 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.694 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.694 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.694 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.386366+0000", 2024-04-26T01:08:49.694 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013135499999999999 2024-04-26T01:08:49.694 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.694 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.695 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.695 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.695 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.695 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/30 ECSubWrite(tid=318, reqid=client.4521.0:601, at_version=58'41, trim_to=0'0, roll_forward_to=58'39))", 2024-04-26T01:08:49.695 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.359815+0000", 2024-04-26T01:08:49.695 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 25.298261918000001, 2024-04-26T01:08:49.695 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.11192218399999999, 2024-04-26T01:08:49.695 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.695 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.695 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:49.695 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.695 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.695 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.695 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.359815+0000", 2024-04-26T01:08:49.696 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.696 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.696 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.696 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.696 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.359815+0000", 2024-04-26T01:08:49.696 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.696 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.696 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.696 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.696 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.359816+0000", 2024-04-26T01:08:49.696 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1429999999999999e-06 2024-04-26T01:08:49.696 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.696 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.696 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.696 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.359875+0000", 2024-04-26T01:08:49.697 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9364000000000003e-05 2024-04-26T01:08:49.697 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.697 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.697 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.697 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.470037+0000", 2024-04-26T01:08:49.697 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.110162076 2024-04-26T01:08:49.697 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.697 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.697 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.697 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.470055+0000", 2024-04-26T01:08:49.697 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7863000000000002e-05 2024-04-26T01:08:49.697 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.697 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.697 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.697 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.470147+0000", 2024-04-26T01:08:49.698 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2103999999999998e-05 2024-04-26T01:08:49.698 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.698 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.698 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:49.698 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.470157+0000", 2024-04-26T01:08:49.698 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0182e-05 2024-04-26T01:08:49.698 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.698 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.698 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:49.698 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.471667+0000", 2024-04-26T01:08:49.698 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015096739999999999 2024-04-26T01:08:49.698 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.698 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.698 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.699 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.471737+0000", 2024-04-26T01:08:49.699 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9777999999999995e-05 2024-04-26T01:08:49.699 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.699 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.699 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.699 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.699 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.699 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.cs2 59/44 ECSubWrite(tid=46, reqid=client.4521.0:635, at_version=59'56, trim_to=0'0, roll_forward_to=59'54))", 2024-04-26T01:08:49.699 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:25.524509+0000", 2024-04-26T01:08:49.699 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 24.133567665000001, 2024-04-26T01:08:49.699 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24557240299999999, 2024-04-26T01:08:49.699 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.699 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.699 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:49.699 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.700 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.700 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.700 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.524509+0000", 2024-04-26T01:08:49.700 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.700 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.700 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.700 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.700 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.524509+0000", 2024-04-26T01:08:49.700 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.700 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.700 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.700 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.700 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.524510+0000", 2024-04-26T01:08:49.700 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4199999999999995e-07 2024-04-26T01:08:49.700 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.701 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.701 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.701 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.524523+0000", 2024-04-26T01:08:49.701 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3491e-05 2024-04-26T01:08:49.701 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.701 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.701 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.701 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.768874+0000", 2024-04-26T01:08:49.701 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24435099299999999 2024-04-26T01:08:49.701 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.701 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.701 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.701 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.768892+0000", 2024-04-26T01:08:49.701 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7886000000000001e-05 2024-04-26T01:08:49.702 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.702 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.702 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.702 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.768997+0000", 2024-04-26T01:08:49.702 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010476000000000001 2024-04-26T01:08:49.702 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.702 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.702 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:49.702 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.769012+0000", 2024-04-26T01:08:49.702 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5641000000000002e-05 2024-04-26T01:08:49.702 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.702 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.702 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:49.702 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.770034+0000", 2024-04-26T01:08:49.702 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001022071 2024-04-26T01:08:49.703 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.703 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.703 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.703 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.770081+0000", 2024-04-26T01:08:49.703 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6919000000000003e-05 2024-04-26T01:08:49.703 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.703 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.703 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.703 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.703 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.703 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:641 3.6s0 3:69323dd0:::smithi16428812-38:head [delete] snapc f=[f] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e60)", 2024-04-26T01:08:49.703 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:26.167403+0000", 2024-04-26T01:08:49.703 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 23.49067354, 2024-04-26T01:08:49.703 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.025490321999999999, 2024-04-26T01:08:49.703 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.704 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.704 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:49.704 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:49.704 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:49.704 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:49.704 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 641 2024-04-26T01:08:49.704 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.704 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.704 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.704 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.704 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167403+0000", 2024-04-26T01:08:49.704 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.704 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.704 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.705 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.705 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167403+0000", 2024-04-26T01:08:49.705 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.705 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.705 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.705 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.705 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167406+0000", 2024-04-26T01:08:49.705 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7920000000000001e-06 2024-04-26T01:08:49.705 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.705 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.705 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.705 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167415+0000", 2024-04-26T01:08:49.705 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1540000000000008e-06 2024-04-26T01:08:49.705 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.705 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.706 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.706 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167442+0000", 2024-04-26T01:08:49.706 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6806999999999999e-05 2024-04-26T01:08:49.706 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.706 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.706 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.706 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167457+0000", 2024-04-26T01:08:49.706 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4836e-05 2024-04-26T01:08:49.706 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.706 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.706 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.706 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167580+0000", 2024-04-26T01:08:49.706 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012298800000000001 2024-04-26T01:08:49.706 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.707 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.707 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op must wait for map", 2024-04-26T01:08:49.707 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.167594+0000", 2024-04-26T01:08:49.707 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4328000000000001e-05 2024-04-26T01:08:49.707 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.707 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.707 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.707 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.172291+0000", 2024-04-26T01:08:49.707 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046972139999999999 2024-04-26T01:08:49.707 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.707 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.707 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:49.707 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.172409+0000", 2024-04-26T01:08:49.707 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000118066 2024-04-26T01:08:49.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.708 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.708 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:49.708 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.173600+0000", 2024-04-26T01:08:49.708 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001190839 2024-04-26T01:08:49.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.708 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.708 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:49.708 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.180944+0000", 2024-04-26T01:08:49.708 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0073435790000000003 2024-04-26T01:08:49.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.708 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.708 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:49.708 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.192802+0000", 2024-04-26T01:08:49.708 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011858848999999999 2024-04-26T01:08:49.709 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.709 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.709 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.709 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:26.192893+0000", 2024-04-26T01:08:49.709 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0870000000000002e-05 2024-04-26T01:08:49.709 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.709 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.709 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.709 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.709 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.709 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpRead(3.7s3 63/44 ECSubRead(tid=373, to_read={3:e163c084:::smithi16428812-22:head=0,491520,0}, subchunks={3:e163c084:::smithi16428812-22:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:49.709 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:29.210385+0000", 2024-04-26T01:08:49.709 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 20.447691359, 2024-04-26T01:08:49.709 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.38767122700000001, 2024-04-26T01:08:49.709 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.710 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.710 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:49.710 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.710 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.710 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.710 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.210385+0000", 2024-04-26T01:08:49.710 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.710 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.710 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.710 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.710 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.210385+0000", 2024-04-26T01:08:49.710 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.710 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.710 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.711 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.711 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.210386+0000", 2024-04-26T01:08:49.711 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3399999999999999e-07 2024-04-26T01:08:49.711 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.711 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.711 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.711 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.210398+0000", 2024-04-26T01:08:49.711 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2553e-05 2024-04-26T01:08:49.711 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.711 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.711 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.711 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.597243+0000", 2024-04-26T01:08:49.711 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.38684432099999999 2024-04-26T01:08:49.711 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.711 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.712 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.712 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.597261+0000", 2024-04-26T01:08:49.712 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7861999999999999e-05 2024-04-26T01:08:49.712 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.712 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.712 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.712 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.597490+0000", 2024-04-26T01:08:49.712 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000228982 2024-04-26T01:08:49.712 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.712 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.712 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.712 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.598056+0000", 2024-04-26T01:08:49.712 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000566875 2024-04-26T01:08:49.712 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.713 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.713 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.713 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.713 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.713 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpRead(3.7s3 63/44 ECSubRead(tid=382, to_read={3:ed7aa414:::smithi16428812-13:head=0,53248,0}, subchunks={3:ed7aa414:::smithi16428812-13:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:49.713 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:29.786784+0000", 2024-04-26T01:08:49.713 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 19.871292814, 2024-04-26T01:08:49.713 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.36638038000000001, 2024-04-26T01:08:49.713 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.713 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.713 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:49.713 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.713 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.713 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.713 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.786784+0000", 2024-04-26T01:08:49.714 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.714 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.714 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.714 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.714 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.786784+0000", 2024-04-26T01:08:49.714 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.714 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.714 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.714 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.714 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.786786+0000", 2024-04-26T01:08:49.714 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0269999999999998e-06 2024-04-26T01:08:49.714 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.714 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.714 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.715 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.786810+0000", 2024-04-26T01:08:49.715 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4260000000000002e-05 2024-04-26T01:08:49.715 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.715 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.715 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.715 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:30.152620+0000", 2024-04-26T01:08:49.715 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.36580987500000001 2024-04-26T01:08:49.715 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.715 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.715 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.715 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:30.152649+0000", 2024-04-26T01:08:49.715 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.864e-05 2024-04-26T01:08:49.715 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.715 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.716 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.716 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:30.152857+0000", 2024-04-26T01:08:49.716 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00020862800000000001 2024-04-26T01:08:49.716 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.716 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.716 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.716 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:30.153164+0000", 2024-04-26T01:08:49.716 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00030695000000000001 2024-04-26T01:08:49.716 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.716 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.716 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.716 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.716 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.716 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:867 3.6s0 3:61b5d9f4:::smithi16428812-15:head [rollback 15] snapc 18=[18,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-04-26T01:08:49.716 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:43.277082+0000", 2024-04-26T01:08:49.717 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 6.3809943560000004, 2024-04-26T01:08:49.717 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.020907973, 2024-04-26T01:08:49.717 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.717 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.717 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:49.717 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:49.717 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:49.717 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:49.717 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 867 2024-04-26T01:08:49.717 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.717 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.717 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.717 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.718 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277082+0000", 2024-04-26T01:08:49.718 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.718 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.718 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.718 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.718 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277082+0000", 2024-04-26T01:08:49.718 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.718 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.718 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.718 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.718 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277083+0000", 2024-04-26T01:08:49.718 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1999999999999999e-07 2024-04-26T01:08:49.718 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.718 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.719 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.719 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277087+0000", 2024-04-26T01:08:49.719 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.25e-06 2024-04-26T01:08:49.719 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.719 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.719 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.719 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277096+0000", 2024-04-26T01:08:49.719 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9609999999999994e-06 2024-04-26T01:08:49.719 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.719 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.719 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.719 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277103+0000", 2024-04-26T01:08:49.719 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.652e-06 2024-04-26T01:08:49.719 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.720 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.720 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.720 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277316+0000", 2024-04-26T01:08:49.720 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000213451 2024-04-26T01:08:49.720 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.720 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.720 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:49.720 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277339+0000", 2024-04-26T01:08:49.720 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3027000000000002e-05 2024-04-26T01:08:49.720 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.720 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.720 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.720 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.288393+0000", 2024-04-26T01:08:49.720 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011053449 2024-04-26T01:08:49.720 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.721 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.721 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:49.721 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.288657+0000", 2024-04-26T01:08:49.721 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026475899999999999 2024-04-26T01:08:49.721 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.721 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.721 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:49.721 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.290338+0000", 2024-04-26T01:08:49.721 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0016803720000000001 2024-04-26T01:08:49.721 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.721 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.721 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:49.721 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.294249+0000", 2024-04-26T01:08:49.721 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0039110819999999998 2024-04-26T01:08:49.722 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.722 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.722 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:49.722 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.297903+0000", 2024-04-26T01:08:49.722 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0036538790000000001 2024-04-26T01:08:49.722 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.722 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.722 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.722 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.297990+0000", 2024-04-26T01:08:49.722 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7471000000000003e-05 2024-04-26T01:08:49.722 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.722 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.722 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.722 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.723 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.723 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:868 3.6s0 3:61b5d9f4:::smithi16428812-15:head [append 0~0] snapc 18=[18,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-04-26T01:08:49.723 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:43.277148+0000", 2024-04-26T01:08:49.723 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 6.380928172, 2024-04-26T01:08:49.723 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.023511603999999998, 2024-04-26T01:08:49.723 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.723 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.723 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:49.723 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:49.723 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:49.723 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:49.723 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 868 2024-04-26T01:08:49.723 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.723 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.724 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.724 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.724 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277148+0000", 2024-04-26T01:08:49.724 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.724 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.724 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.724 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.724 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277148+0000", 2024-04-26T01:08:49.724 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.724 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.724 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.724 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.724 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277150+0000", 2024-04-26T01:08:49.724 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.271e-06 2024-04-26T01:08:49.724 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.725 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.725 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.725 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277157+0000", 2024-04-26T01:08:49.725 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7170000000000008e-06 2024-04-26T01:08:49.725 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.725 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.725 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.725 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277176+0000", 2024-04-26T01:08:49.725 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8306999999999999e-05 2024-04-26T01:08:49.725 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.725 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.725 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.725 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277187+0000", 2024-04-26T01:08:49.726 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1620999999999999e-05 2024-04-26T01:08:49.726 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.726 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.726 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.726 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277392+0000", 2024-04-26T01:08:49.726 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00020518500000000001 2024-04-26T01:08:49.726 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.726 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.726 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:49.726 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.277400+0000", 2024-04-26T01:08:49.726 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.379e-06 2024-04-26T01:08:49.726 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.726 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.726 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.727 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.292885+0000", 2024-04-26T01:08:49.727 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015484955 2024-04-26T01:08:49.727 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.727 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.727 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:49.727 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.293029+0000", 2024-04-26T01:08:49.727 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014463200000000001 2024-04-26T01:08:49.727 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.727 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.727 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:49.727 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.293690+0000", 2024-04-26T01:08:49.727 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00066096199999999996 2024-04-26T01:08:49.727 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.728 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.728 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:49.728 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.294669+0000", 2024-04-26T01:08:49.728 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00097881800000000001 2024-04-26T01:08:49.728 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.728 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.728 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:49.728 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.300585+0000", 2024-04-26T01:08:49.728 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.005915554 2024-04-26T01:08:49.728 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.728 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.728 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.728 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:43.300660+0000", 2024-04-26T01:08:49.728 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5203000000000005e-05 2024-04-26T01:08:49.729 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.729 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.729 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.729 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.729 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.729 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 81/30 ECSubWriteReply(tid=95, last_complete=81'65, committed=1, applied=1))", 2024-04-26T01:08:49.729 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:47.330600+0000", 2024-04-26T01:08:49.729 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.3274766659999999, 2024-04-26T01:08:49.729 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.150670375, 2024-04-26T01:08:49.729 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.729 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.729 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:49.729 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.729 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.730 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.730 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.330600+0000", 2024-04-26T01:08:49.730 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.730 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.730 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.730 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.730 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.330600+0000", 2024-04-26T01:08:49.730 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.730 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.730 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.730 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.730 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.330601+0000", 2024-04-26T01:08:49.730 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.155e-06 2024-04-26T01:08:49.730 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.731 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.731 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.731 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.330608+0000", 2024-04-26T01:08:49.731 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1670000000000002e-06 2024-04-26T01:08:49.731 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.731 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.731 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.731 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.480757+0000", 2024-04-26T01:08:49.731 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.15014866599999999 2024-04-26T01:08:49.731 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.731 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.731 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.731 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.480764+0000", 2024-04-26T01:08:49.731 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.703e-06 2024-04-26T01:08:49.732 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.732 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.732 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.732 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.481253+0000", 2024-04-26T01:08:49.732 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000489023 2024-04-26T01:08:49.732 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.732 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.732 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.732 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.481270+0000", 2024-04-26T01:08:49.732 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7660999999999999e-05 2024-04-26T01:08:49.732 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.732 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.732 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.732 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.733 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.733 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:894 3.6s0 3:61b5d9f4:::smithi16428812-15:head [append 0~0] snapc 1b=[1b,1a,19,18] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:08:49.733 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:47.477204+0000", 2024-04-26T01:08:49.733 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.1808729809999998, 2024-04-26T01:08:49.733 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.28359361700000002, 2024-04-26T01:08:49.733 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.733 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.733 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:49.733 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:49.733 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:49.733 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:49.733 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 894 2024-04-26T01:08:49.733 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.734 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.734 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.734 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.734 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.477204+0000", 2024-04-26T01:08:49.734 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.734 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.734 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.734 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.734 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.477204+0000", 2024-04-26T01:08:49.734 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.734 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.734 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.734 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.734 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.477204+0000", 2024-04-26T01:08:49.735 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9699999999999996e-07 2024-04-26T01:08:49.735 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.735 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.735 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.735 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.477207+0000", 2024-04-26T01:08:49.735 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9349999999999999e-06 2024-04-26T01:08:49.735 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.735 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.735 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.735 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.477214+0000", 2024-04-26T01:08:49.735 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9070000000000001e-06 2024-04-26T01:08:49.735 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.735 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.735 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.736 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.477219+0000", 2024-04-26T01:08:49.736 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9899999999999997e-06 2024-04-26T01:08:49.736 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.736 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.736 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.736 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.480348+0000", 2024-04-26T01:08:49.736 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0031294370000000001 2024-04-26T01:08:49.736 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.736 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.736 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:49.736 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.480416+0000", 2024-04-26T01:08:49.736 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7675999999999994e-05 2024-04-26T01:08:49.736 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.737 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.737 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:49.737 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.480852+0000", 2024-04-26T01:08:49.737 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00043568900000000001 2024-04-26T01:08:49.737 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.737 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.737 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:49.737 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.486709+0000", 2024-04-26T01:08:49.737 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0058571279999999996 2024-04-26T01:08:49.737 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.737 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.737 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:49.737 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.760673+0000", 2024-04-26T01:08:49.737 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.273964288 2024-04-26T01:08:49.737 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.738 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.738 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.738 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.760797+0000", 2024-04-26T01:08:49.738 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012407 2024-04-26T01:08:49.738 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.738 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.738 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.738 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.738 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.738 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 81/30 ECSubWriteReply(tid=98, last_complete=81'69, committed=1, applied=1))", 2024-04-26T01:08:49.738 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:47.490166+0000", 2024-04-26T01:08:49.738 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.167910408, 2024-04-26T01:08:49.738 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.27187226799999997, 2024-04-26T01:08:49.738 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.738 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.739 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:49.739 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.739 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.739 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.739 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.490166+0000", 2024-04-26T01:08:49.739 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.739 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.739 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.739 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.739 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.490166+0000", 2024-04-26T01:08:49.739 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.739 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.739 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.739 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.739 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.490167+0000", 2024-04-26T01:08:49.740 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0699999999999997e-07 2024-04-26T01:08:49.740 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.740 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.740 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.740 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.490170+0000", 2024-04-26T01:08:49.740 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.394e-06 2024-04-26T01:08:49.740 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.740 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.740 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.740 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.760360+0000", 2024-04-26T01:08:49.740 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.27019027099999998 2024-04-26T01:08:49.740 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.740 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.740 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.741 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.760383+0000", 2024-04-26T01:08:49.741 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2140000000000001e-05 2024-04-26T01:08:49.741 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.741 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.741 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.741 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.760537+0000", 2024-04-26T01:08:49.741 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015395599999999999 2024-04-26T01:08:49.741 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.741 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.741 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.741 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.762038+0000", 2024-04-26T01:08:49.741 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015019 2024-04-26T01:08:49.741 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.741 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.742 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.742 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.742 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.742 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.5s5 81/30 ECSubWrite(tid=69, reqid=client.4521.0:907, at_version=81'56, trim_to=0'0, roll_forward_to=81'54))", 2024-04-26T01:08:49.742 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:47.502880+0000", 2024-04-26T01:08:49.742 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.1551969099999999, 2024-04-26T01:08:49.742 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.20784002500000001, 2024-04-26T01:08:49.742 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.742 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.742 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:49.742 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.742 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.742 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.743 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.502880+0000", 2024-04-26T01:08:49.743 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.743 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.743 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.743 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.743 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.502880+0000", 2024-04-26T01:08:49.743 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.743 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.743 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.743 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.743 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.502881+0000", 2024-04-26T01:08:49.743 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4339999999999999e-06 2024-04-26T01:08:49.743 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.743 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.744 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.744 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.502905+0000", 2024-04-26T01:08:49.744 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4071999999999999e-05 2024-04-26T01:08:49.744 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.744 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.744 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.744 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.707930+0000", 2024-04-26T01:08:49.744 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.20502498799999999 2024-04-26T01:08:49.744 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.744 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.744 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.744 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.707946+0000", 2024-04-26T01:08:49.744 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5466999999999999e-05 2024-04-26T01:08:49.744 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.744 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.745 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.745 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.708113+0000", 2024-04-26T01:08:49.745 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016742300000000001 2024-04-26T01:08:49.745 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.745 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.745 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:49.745 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.708146+0000", 2024-04-26T01:08:49.745 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.2699e-05 2024-04-26T01:08:49.745 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.745 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.745 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:49.745 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.710671+0000", 2024-04-26T01:08:49.745 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0025253559999999999 2024-04-26T01:08:49.745 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.745 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.746 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.746 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.710720+0000", 2024-04-26T01:08:49.746 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8585999999999999e-05 2024-04-26T01:08:49.746 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.746 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.746 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.746 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.746 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.746 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.fs6 82/44 ECSubWrite(tid=489, reqid=client.4521.0:910, at_version=82'111, trim_to=0'0, roll_forward_to=81'108))", 2024-04-26T01:08:49.746 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:48.329338+0000", 2024-04-26T01:08:49.746 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.3287385810000001, 2024-04-26T01:08:49.746 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.025602127999999998, 2024-04-26T01:08:49.746 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.746 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.747 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:49.747 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.747 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.747 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.747 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.329338+0000", 2024-04-26T01:08:49.747 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.747 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.747 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.747 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.747 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.329338+0000", 2024-04-26T01:08:49.747 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.747 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.747 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.748 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.748 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.329339+0000", 2024-04-26T01:08:49.748 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3200000000000003e-07 2024-04-26T01:08:49.748 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.748 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.748 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.748 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.329369+0000", 2024-04-26T01:08:49.748 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0422000000000001e-05 2024-04-26T01:08:49.748 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.748 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.748 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.748 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.353456+0000", 2024-04-26T01:08:49.748 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.024086171999999999 2024-04-26T01:08:49.748 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.353469+0000", 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3485999999999999e-05 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.353610+0000", 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000141046 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.353631+0000", 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0993e-05 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.749 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.750 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:49.750 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.354886+0000", 2024-04-26T01:08:49.750 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0012544800000000001 2024-04-26T01:08:49.750 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.750 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.750 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.750 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.354940+0000", 2024-04-26T01:08:49.750 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4596999999999999e-05 2024-04-26T01:08:49.750 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.750 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.750 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.750 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.750 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.750 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.bs1 82/30 ECSubWrite(tid=493, reqid=client.4521.0:912, at_version=0'0, trim_to=0'0, roll_forward_to=82'73))", 2024-04-26T01:08:49.751 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:48.360908+0000", 2024-04-26T01:08:49.751 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.2971687780000001, 2024-04-26T01:08:49.751 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.77112417300000002, 2024-04-26T01:08:49.751 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:49.751 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:49.751 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:49.751 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:49.751 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.751 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:49.751 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.360908+0000", 2024-04-26T01:08:49.751 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.751 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.751 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.751 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:49.751 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.360908+0000", 2024-04-26T01:08:49.752 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:49.752 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.752 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.752 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:49.752 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.360909+0000", 2024-04-26T01:08:49.752 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.218e-06 2024-04-26T01:08:49.752 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.752 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.752 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:49.752 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.360925+0000", 2024-04-26T01:08:49.752 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5829e-05 2024-04-26T01:08:49.752 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.752 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.752 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:49.753 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.130748+0000", 2024-04-26T01:08:49.753 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.76982344899999999 2024-04-26T01:08:49.753 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.753 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.753 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:49.753 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.130766+0000", 2024-04-26T01:08:49.753 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7486000000000002e-05 2024-04-26T01:08:49.753 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.753 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.753 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:49.753 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.130930+0000", 2024-04-26T01:08:49.753 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016400299999999999 2024-04-26T01:08:49.753 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.753 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.753 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:49.754 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.130950+0000", 2024-04-26T01:08:49.754 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9956e-05 2024-04-26T01:08:49.754 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.754 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.754 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:49.754 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.131956+0000", 2024-04-26T01:08:49.754 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0010063229999999999 2024-04-26T01:08:49.754 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:49.754 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:49.754 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:49.754 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.132032+0000", 2024-04-26T01:08:49.754 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5908999999999994e-05 2024-04-26T01:08:49.754 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.754 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.754 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.755 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:49.755 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:49.755 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:49.756 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:08:49.757 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:49.755+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:49.858 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:49.855+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:49.887 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:49.887 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:49.887 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-04-26T01:08:49.887 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:49.903 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:08:49.959 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:49.957+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:50.030 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:50.030 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:50.030 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-04-26T01:08:50.030 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-04-26T01:08:50.030 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:50.044 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:08:50.089 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:50.088+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:50.161 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:50.157+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:50.172 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:50.172 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:50.172 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:50.172 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:50.172 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.172 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.5s2 37/30 ECSubWrite(tid=22, reqid=client.4521.0:251, at_version=37'17, trim_to=0'0, roll_forward_to=36'13))", 2024-04-26T01:08:50.172 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:03.434740+0000", 2024-04-26T01:08:50.173 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 46.734769082, 2024-04-26T01:08:50.173 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.47257335099999997, 2024-04-26T01:08:50.173 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.173 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.173 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.173 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.173 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.173 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.173 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434740+0000", 2024-04-26T01:08:50.173 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.173 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.173 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.173 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.173 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434740+0000", 2024-04-26T01:08:50.174 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.174 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.174 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.174 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.174 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434741+0000", 2024-04-26T01:08:50.174 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8800000000000002e-07 2024-04-26T01:08:50.174 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.174 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.174 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.174 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.434764+0000", 2024-04-26T01:08:50.174 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3499999999999999e-05 2024-04-26T01:08:50.174 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.174 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.174 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.175 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906023+0000", 2024-04-26T01:08:50.175 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.47125867799999999 2024-04-26T01:08:50.175 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.175 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.175 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.175 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906041+0000", 2024-04-26T01:08:50.175 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7887e-05 2024-04-26T01:08:50.175 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.175 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.175 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.175 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906172+0000", 2024-04-26T01:08:50.175 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013065200000000001 2024-04-26T01:08:50.175 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.175 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.176 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:50.176 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.906192+0000", 2024-04-26T01:08:50.176 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0363000000000001e-05 2024-04-26T01:08:50.176 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.176 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.176 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:50.176 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.907240+0000", 2024-04-26T01:08:50.176 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0010479 2024-04-26T01:08:50.176 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.176 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.176 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.176 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:03.907314+0000", 2024-04-26T01:08:50.176 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3583000000000003e-05 2024-04-26T01:08:50.176 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.177 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.177 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.177 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.177 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.177 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 52/30 ECSubWriteReply(tid=246, last_complete=52'26, committed=1, applied=1))", 2024-04-26T01:08:50.177 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:18.467431+0000", 2024-04-26T01:08:50.177 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 31.702078159999999, 2024-04-26T01:08:50.177 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.59825841400000002, 2024-04-26T01:08:50.177 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.177 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.177 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.177 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.177 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.178 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.178 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:18.467431+0000", 2024-04-26T01:08:50.178 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.178 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.178 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.178 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.178 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:18.467431+0000", 2024-04-26T01:08:50.178 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.178 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.178 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.178 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.178 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:18.467432+0000", 2024-04-26T01:08:50.178 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5400000000000001e-07 2024-04-26T01:08:50.178 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.179 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.179 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.179 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:18.467437+0000", 2024-04-26T01:08:50.179 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1200000000000001e-06 2024-04-26T01:08:50.179 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.179 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.179 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.179 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.065059+0000", 2024-04-26T01:08:50.179 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.59762205800000001 2024-04-26T01:08:50.179 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.179 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.179 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.179 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.065068+0000", 2024-04-26T01:08:50.179 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9430000000000006e-06 2024-04-26T01:08:50.179 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.180 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.180 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.180 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.065136+0000", 2024-04-26T01:08:50.180 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7841000000000003e-05 2024-04-26T01:08:50.180 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.180 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.180 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.180 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:19.065690+0000", 2024-04-26T01:08:50.180 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00055379799999999999 2024-04-26T01:08:50.180 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.180 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.180 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.180 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.180 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.181 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:601 3.bs0 3:d5691d7a:::smithi16428812-40:head [copy-from ver 4] snapc f=[f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:50.181 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.340640+0000", 2024-04-26T01:08:50.181 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 25.828869298000001, 2024-04-26T01:08:50.181 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.78325740899999996, 2024-04-26T01:08:50.181 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.181 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.181 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:50.181 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.181 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.181 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.181 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 601 2024-04-26T01:08:50.181 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.181 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.181 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.182 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.182 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340640+0000", 2024-04-26T01:08:50.182 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.182 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.182 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.182 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.182 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340640+0000", 2024-04-26T01:08:50.182 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.182 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.182 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.182 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.182 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340644+0000", 2024-04-26T01:08:50.182 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9169999999999999e-06 2024-04-26T01:08:50.182 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.182 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.183 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.183 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340661+0000", 2024-04-26T01:08:50.183 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7390000000000001e-05 2024-04-26T01:08:50.183 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.183 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.183 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.183 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340696+0000", 2024-04-26T01:08:50.183 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4332999999999998e-05 2024-04-26T01:08:50.183 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.183 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.183 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.183 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340728+0000", 2024-04-26T01:08:50.183 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.2246000000000003e-05 2024-04-26T01:08:50.183 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.184 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.184 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.184 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340944+0000", 2024-04-26T01:08:50.184 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00021597700000000001 2024-04-26T01:08:50.184 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.184 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.184 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:50.184 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341149+0000", 2024-04-26T01:08:50.184 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000204725 2024-04-26T01:08:50.184 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.184 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.184 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:50.184 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.360012+0000", 2024-04-26T01:08:50.184 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018863683999999999 2024-04-26T01:08:50.185 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.185 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.185 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:50.185 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.362478+0000", 2024-04-26T01:08:50.185 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024654849999999999 2024-04-26T01:08:50.185 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.185 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.185 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:50.185 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.123741+0000", 2024-04-26T01:08:50.185 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.76126282099999998 2024-04-26T01:08:50.185 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.185 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.185 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.185 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.123897+0000", 2024-04-26T01:08:50.185 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015683099999999999 2024-04-26T01:08:50.186 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.186 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.186 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.186 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.186 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.186 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 62/44 ECSubWriteReply(tid=370, last_complete=63'73, committed=1, applied=1))", 2024-04-26T01:08:50.186 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:29.200166+0000", 2024-04-26T01:08:50.186 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 20.969343335000001, 2024-04-26T01:08:50.186 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.42031032000000002, 2024-04-26T01:08:50.186 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.186 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.186 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.186 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.187 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.187 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.187 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.200166+0000", 2024-04-26T01:08:50.187 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.187 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.187 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.187 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.187 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.200166+0000", 2024-04-26T01:08:50.187 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.187 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.187 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.187 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.187 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.200168+0000", 2024-04-26T01:08:50.187 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0370000000000001e-06 2024-04-26T01:08:50.188 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.188 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.188 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.188 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.200180+0000", 2024-04-26T01:08:50.188 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1633e-05 2024-04-26T01:08:50.188 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.188 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.188 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.188 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.619636+0000", 2024-04-26T01:08:50.188 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.41945664399999999 2024-04-26T01:08:50.188 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.188 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.188 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.188 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.619652+0000", 2024-04-26T01:08:50.189 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5673000000000001e-05 2024-04-26T01:08:50.189 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.189 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.189 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.189 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.619741+0000", 2024-04-26T01:08:50.189 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9479000000000002e-05 2024-04-26T01:08:50.189 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.189 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.189 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.189 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.620476+0000", 2024-04-26T01:08:50.189 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00073485399999999995 2024-04-26T01:08:50.189 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.189 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.189 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.190 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.190 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.190 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:688 3.7s0 3:ed7aa414:::smithi16428812-13:head [read 0~212992,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-04-26T01:08:50.190 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:29.783527+0000", 2024-04-26T01:08:50.190 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 20.385982128999999, 2024-04-26T01:08:50.190 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.37062857599999999, 2024-04-26T01:08:50.190 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.190 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.190 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:50.190 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.190 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.190 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.190 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 688 2024-04-26T01:08:50.190 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.191 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.191 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.191 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.191 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.783527+0000", 2024-04-26T01:08:50.191 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.191 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.191 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.191 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.191 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.783527+0000", 2024-04-26T01:08:50.191 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.191 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.191 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.191 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.191 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.783528+0000", 2024-04-26T01:08:50.192 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.187e-06 2024-04-26T01:08:50.192 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.192 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.192 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.192 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.783535+0000", 2024-04-26T01:08:50.192 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3130000000000004e-06 2024-04-26T01:08:50.192 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.192 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.192 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.192 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.783546+0000", 2024-04-26T01:08:50.192 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1093e-05 2024-04-26T01:08:50.192 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.192 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.192 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.193 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.783553+0000", 2024-04-26T01:08:50.193 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6899999999999992e-06 2024-04-26T01:08:50.193 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.193 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.193 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.193 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.785428+0000", 2024-04-26T01:08:50.193 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001874697 2024-04-26T01:08:50.193 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.193 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.193 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:50.193 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:29.785679+0000", 2024-04-26T01:08:50.193 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00025062800000000003 2024-04-26T01:08:50.193 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.193 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.194 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.194 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:30.154156+0000", 2024-04-26T01:08:50.194 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.36847696800000002 2024-04-26T01:08:50.194 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.194 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.194 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.194 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.194 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.194 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:762 3.fs0 3:fcac9f29:::smithi16428812-36:head [setxattr _header (57) in=7b] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:08:50.194 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.224739+0000", 2024-04-26T01:08:50.194 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.944770694000001, 2024-04-26T01:08:50.194 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35577256200000001, 2024-04-26T01:08:50.194 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.194 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.195 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:50.195 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.195 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.195 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.195 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 762 2024-04-26T01:08:50.195 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.195 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.195 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.195 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.195 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224739+0000", 2024-04-26T01:08:50.195 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.195 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.195 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.195 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.196 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224739+0000", 2024-04-26T01:08:50.196 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.196 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.196 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.196 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.196 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224739+0000", 2024-04-26T01:08:50.196 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3e-07 2024-04-26T01:08:50.196 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.196 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.196 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.196 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224744+0000", 2024-04-26T01:08:50.196 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5920000000000002e-06 2024-04-26T01:08:50.196 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.196 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.197 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.197 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224751+0000", 2024-04-26T01:08:50.197 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6039999999999998e-06 2024-04-26T01:08:50.197 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.197 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.197 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.197 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224756+0000", 2024-04-26T01:08:50.197 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.006e-06 2024-04-26T01:08:50.197 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.197 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.197 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.197 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.228205+0000", 2024-04-26T01:08:50.197 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003449971 2024-04-26T01:08:50.197 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.198 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.198 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:50.198 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.228282+0000", 2024-04-26T01:08:50.198 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6625000000000006e-05 2024-04-26T01:08:50.198 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.198 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.198 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:50.198 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.228828+0000", 2024-04-26T01:08:50.198 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00054617499999999998 2024-04-26T01:08:50.198 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.198 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.198 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:50.198 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.230451+0000", 2024-04-26T01:08:50.199 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001622477 2024-04-26T01:08:50.199 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.199 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.199 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:50.199 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.580252+0000", 2024-04-26T01:08:50.199 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.34980141599999998 2024-04-26T01:08:50.199 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.199 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.199 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.199 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.580511+0000", 2024-04-26T01:08:50.199 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00025896600000000001 2024-04-26T01:08:50.199 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.199 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.199 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.200 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.200 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.200 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:763 3.fs0 3:fcac9f29:::smithi16428812-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:08:50.200 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.224763+0000", 2024-04-26T01:08:50.200 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.944746144, 2024-04-26T01:08:50.200 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35969443000000001, 2024-04-26T01:08:50.200 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.200 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.200 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:50.200 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.200 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.200 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.200 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 763 2024-04-26T01:08:50.200 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.201 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.201 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.201 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.201 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224763+0000", 2024-04-26T01:08:50.201 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.201 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.201 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.201 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.201 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224763+0000", 2024-04-26T01:08:50.201 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.201 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.201 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.201 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.201 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224764+0000", 2024-04-26T01:08:50.202 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8999999999999996e-07 2024-04-26T01:08:50.202 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.202 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.202 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.202 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224767+0000", 2024-04-26T01:08:50.202 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.118e-06 2024-04-26T01:08:50.202 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.202 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.202 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.202 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224774+0000", 2024-04-26T01:08:50.202 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0840000000000003e-06 2024-04-26T01:08:50.202 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.202 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.202 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.203 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.224779+0000", 2024-04-26T01:08:50.203 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.31e-06 2024-04-26T01:08:50.203 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.203 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.203 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.203 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.229405+0000", 2024-04-26T01:08:50.203 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046254920000000001 2024-04-26T01:08:50.203 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.203 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.203 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:50.203 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.229732+0000", 2024-04-26T01:08:50.203 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00032752299999999997 2024-04-26T01:08:50.203 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.203 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.204 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.204 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.581793+0000", 2024-04-26T01:08:50.204 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35206099899999999 2024-04-26T01:08:50.204 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.204 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.204 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:50.204 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.582075+0000", 2024-04-26T01:08:50.204 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00028167900000000001 2024-04-26T01:08:50.204 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.204 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.204 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.204 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.584458+0000", 2024-04-26T01:08:50.204 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0023826350000000001 2024-04-26T01:08:50.204 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.205 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.205 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.205 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.205 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.205 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 71/44 ECSubWriteReply(tid=420, last_complete=71'100, committed=1, applied=1))", 2024-04-26T01:08:50.205 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.229879+0000", 2024-04-26T01:08:50.205 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.939630266, 2024-04-26T01:08:50.205 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35171478499999997, 2024-04-26T01:08:50.205 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.205 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.205 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.205 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.205 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.205 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.205 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.229879+0000", 2024-04-26T01:08:50.206 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.206 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.206 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.206 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.206 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.229879+0000", 2024-04-26T01:08:50.206 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.206 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.206 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.206 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.206 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.229880+0000", 2024-04-26T01:08:50.206 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1960000000000001e-06 2024-04-26T01:08:50.206 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.206 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.206 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.207 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.229888+0000", 2024-04-26T01:08:50.207 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5399999999999998e-06 2024-04-26T01:08:50.207 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.207 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.207 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.207 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.579761+0000", 2024-04-26T01:08:50.207 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.34987286400000001 2024-04-26T01:08:50.207 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.207 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.207 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.207 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.579780+0000", 2024-04-26T01:08:50.207 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9147000000000002e-05 2024-04-26T01:08:50.207 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.207 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.208 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.208 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.579993+0000", 2024-04-26T01:08:50.208 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000213599 2024-04-26T01:08:50.208 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.208 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.208 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.208 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.581594+0000", 2024-04-26T01:08:50.208 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0016004389999999999 2024-04-26T01:08:50.208 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.208 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.208 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.208 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.208 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.208 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:768 3.bs0 3:d8d6d69b:::smithi16428812-48:head [setxattr _header (57) in=7b] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:08:50.208 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.237741+0000", 2024-04-26T01:08:50.209 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.931768400999999, 2024-04-26T01:08:50.209 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.36214788399999998, 2024-04-26T01:08:50.209 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.209 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.209 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:50.209 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.209 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.209 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.209 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 768 2024-04-26T01:08:50.209 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.209 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.209 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.209 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.210 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237741+0000", 2024-04-26T01:08:50.210 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.210 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.210 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.210 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.210 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237741+0000", 2024-04-26T01:08:50.210 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.210 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.210 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.210 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.210 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237742+0000", 2024-04-26T01:08:50.210 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4200000000000005e-07 2024-04-26T01:08:50.210 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.210 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.211 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.211 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237747+0000", 2024-04-26T01:08:50.211 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.921e-06 2024-04-26T01:08:50.211 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.211 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.211 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.211 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237757+0000", 2024-04-26T01:08:50.211 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0068000000000001e-05 2024-04-26T01:08:50.211 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.211 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.211 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.211 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237764+0000", 2024-04-26T01:08:50.211 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9689999999999997e-06 2024-04-26T01:08:50.211 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.211 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.212 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.212 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.241404+0000", 2024-04-26T01:08:50.212 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0036398590000000001 2024-04-26T01:08:50.212 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.212 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.212 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:50.212 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.241528+0000", 2024-04-26T01:08:50.212 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000124258 2024-04-26T01:08:50.212 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.212 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.212 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:50.212 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.242111+0000", 2024-04-26T01:08:50.212 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00058331400000000003 2024-04-26T01:08:50.212 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.213 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.213 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:50.213 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.242834+0000", 2024-04-26T01:08:50.213 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00072257799999999998 2024-04-26T01:08:50.213 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.213 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.213 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:50.213 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.599608+0000", 2024-04-26T01:08:50.213 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35677461399999999 2024-04-26T01:08:50.213 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.213 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.213 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.213 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.599889+0000", 2024-04-26T01:08:50.213 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00028046100000000002 2024-04-26T01:08:50.213 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.214 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.214 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.214 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.214 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.214 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:769 3.bs0 3:d8d6d69b:::smithi16428812-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:08:50.214 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.237777+0000", 2024-04-26T01:08:50.214 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.931732326000001, 2024-04-26T01:08:50.214 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.840927439, 2024-04-26T01:08:50.214 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.214 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.214 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:50.214 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.214 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.214 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.215 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 769 2024-04-26T01:08:50.215 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.215 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.215 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.215 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.215 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237777+0000", 2024-04-26T01:08:50.215 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.215 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.215 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.215 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.215 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237777+0000", 2024-04-26T01:08:50.215 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.215 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.215 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.216 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.216 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237778+0000", 2024-04-26T01:08:50.216 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.92e-07 2024-04-26T01:08:50.216 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.216 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.216 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.216 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237782+0000", 2024-04-26T01:08:50.216 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7029999999999997e-06 2024-04-26T01:08:50.216 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.216 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.216 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.216 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237789+0000", 2024-04-26T01:08:50.216 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.063e-06 2024-04-26T01:08:50.216 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.217 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.217 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.217 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.237796+0000", 2024-04-26T01:08:50.217 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1330000000000004e-06 2024-04-26T01:08:50.217 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.217 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.217 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.217 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.242751+0000", 2024-04-26T01:08:50.217 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0049552069999999997 2024-04-26T01:08:50.217 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.217 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.217 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:50.217 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.242818+0000", 2024-04-26T01:08:50.218 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7502999999999994e-05 2024-04-26T01:08:50.218 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.218 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.218 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.218 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.601011+0000", 2024-04-26T01:08:50.218 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.358193066 2024-04-26T01:08:50.218 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.218 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.218 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:50.218 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.601277+0000", 2024-04-26T01:08:50.218 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026577300000000002 2024-04-26T01:08:50.218 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.218 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.218 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.219 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.078704+0000", 2024-04-26T01:08:50.219 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.47742729900000003 2024-04-26T01:08:50.219 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.219 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.219 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.219 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.219 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.219 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 71/30 ECSubWriteReply(tid=425, last_complete=71'63, committed=1, applied=1))", 2024-04-26T01:08:50.219 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.243053+0000", 2024-04-26T01:08:50.219 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.926456343, 2024-04-26T01:08:50.219 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35781853899999999, 2024-04-26T01:08:50.219 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.219 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.219 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.220 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.220 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.220 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.220 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.243053+0000", 2024-04-26T01:08:50.220 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.220 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.220 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.220 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.220 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.243053+0000", 2024-04-26T01:08:50.220 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.220 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.220 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.220 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.220 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.243053+0000", 2024-04-26T01:08:50.221 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.0800000000000005e-07 2024-04-26T01:08:50.221 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.221 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.221 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.221 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.243057+0000", 2024-04-26T01:08:50.221 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4309999999999998e-06 2024-04-26T01:08:50.221 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.221 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.221 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.221 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.598402+0000", 2024-04-26T01:08:50.221 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35534554099999999 2024-04-26T01:08:50.221 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.221 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.221 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.221 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.598415+0000", 2024-04-26T01:08:50.222 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.279e-05 2024-04-26T01:08:50.222 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.222 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.222 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.222 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.599361+0000", 2024-04-26T01:08:50.222 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00094610600000000005 2024-04-26T01:08:50.222 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.222 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.222 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.222 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.600871+0000", 2024-04-26T01:08:50.222 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015101629999999999 2024-04-26T01:08:50.222 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.222 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.223 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.223 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.223 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.223 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 71/44 ECSubWriteReply(tid=426, last_complete=71'100, committed=1, applied=1))", 2024-04-26T01:08:50.223 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.582864+0000", 2024-04-26T01:08:50.223 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.586645703, 2024-04-26T01:08:50.223 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.42299990500000001, 2024-04-26T01:08:50.223 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.223 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.223 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.223 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.223 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.223 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.223 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.582864+0000", 2024-04-26T01:08:50.223 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.224 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.224 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.224 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.224 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.582864+0000", 2024-04-26T01:08:50.224 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.224 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.224 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.224 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.224 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.582865+0000", 2024-04-26T01:08:50.224 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0899999999999999e-06 2024-04-26T01:08:50.224 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.224 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.224 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.224 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.582874+0000", 2024-04-26T01:08:50.225 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9360000000000005e-06 2024-04-26T01:08:50.225 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.225 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.225 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.225 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.005609+0000", 2024-04-26T01:08:50.225 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.422735317 2024-04-26T01:08:50.225 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.225 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.225 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.225 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.005623+0000", 2024-04-26T01:08:50.225 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.394e-05 2024-04-26T01:08:50.225 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.225 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.225 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.226 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.005793+0000", 2024-04-26T01:08:50.226 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000170142 2024-04-26T01:08:50.226 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.226 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.226 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.226 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.005863+0000", 2024-04-26T01:08:50.226 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0480000000000003e-05 2024-04-26T01:08:50.226 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.226 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.226 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.226 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.226 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.226 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 71/30 ECSubWriteReply(tid=428, last_complete=71'63, committed=1, applied=1))", 2024-04-26T01:08:50.227 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.601257+0000", 2024-04-26T01:08:50.227 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.568252356, 2024-04-26T01:08:50.227 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.99078997400000002, 2024-04-26T01:08:50.227 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.227 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.227 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.227 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.227 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.227 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.601257+0000", 2024-04-26T01:08:50.227 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.227 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.227 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.601257+0000", 2024-04-26T01:08:50.228 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.228 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.228 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.228 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.601259+0000", 2024-04-26T01:08:50.228 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.59e-06 2024-04-26T01:08:50.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.228 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.228 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.228 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.601268+0000", 2024-04-26T01:08:50.228 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.38e-06 2024-04-26T01:08:50.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.228 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.228 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.229 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.591752+0000", 2024-04-26T01:08:50.229 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.990484162 2024-04-26T01:08:50.229 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.229 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.229 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.229 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.591768+0000", 2024-04-26T01:08:50.229 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6169999999999999e-05 2024-04-26T01:08:50.229 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.229 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.229 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.229 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.591964+0000", 2024-04-26T01:08:50.229 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019577900000000001 2024-04-26T01:08:50.229 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.229 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.229 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.230 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.592047+0000", 2024-04-26T01:08:50.230 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2892999999999994e-05 2024-04-26T01:08:50.230 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.230 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.230 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.230 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.230 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.230 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpReadReply(3.bs0 71/30 ECSubReadReply(tid=429, attrs_read=0))", 2024-04-26T01:08:50.230 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:37.602946+0000", 2024-04-26T01:08:50.230 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.566563488, 2024-04-26T01:08:50.230 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.47577750400000002, 2024-04-26T01:08:50.230 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.230 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.230 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.231 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.231 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.231 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.602946+0000", 2024-04-26T01:08:50.231 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.231 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.231 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.231 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.602946+0000", 2024-04-26T01:08:50.231 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.231 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.231 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.231 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.602947+0000", 2024-04-26T01:08:50.232 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2699999999999998e-07 2024-04-26T01:08:50.232 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.232 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.232 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:37.602961+0000", 2024-04-26T01:08:50.232 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.485e-05 2024-04-26T01:08:50.232 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.232 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.232 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.078257+0000", 2024-04-26T01:08:50.232 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.47529542299999999 2024-04-26T01:08:50.232 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.232 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.233 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.078275+0000", 2024-04-26T01:08:50.233 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7796e-05 2024-04-26T01:08:50.233 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.233 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.233 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.233 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.078412+0000", 2024-04-26T01:08:50.233 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013743300000000001 2024-04-26T01:08:50.233 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.233 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.233 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.233 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:38.078723+0000", 2024-04-26T01:08:50.233 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00031117500000000001 2024-04-26T01:08:50.233 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.233 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.234 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.234 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.234 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.234 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 81/44 ECSubWriteReply(tid=473, last_complete=81'106, committed=1, applied=1))", 2024-04-26T01:08:50.234 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:47.346561+0000", 2024-04-26T01:08:50.234 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.822948475, 2024-04-26T01:08:50.234 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.15084829399999999, 2024-04-26T01:08:50.234 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.234 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.234 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.234 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.234 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.234 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.234 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.346561+0000", 2024-04-26T01:08:50.234 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.235 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.235 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.235 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.235 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.346561+0000", 2024-04-26T01:08:50.235 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.235 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.235 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.235 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.235 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.346562+0000", 2024-04-26T01:08:50.235 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.395e-06 2024-04-26T01:08:50.235 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.235 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.235 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.235 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.346569+0000", 2024-04-26T01:08:50.236 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.596e-06 2024-04-26T01:08:50.236 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.236 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.236 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.236 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.497307+0000", 2024-04-26T01:08:50.236 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.15073832200000001 2024-04-26T01:08:50.236 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.236 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.236 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.236 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.497315+0000", 2024-04-26T01:08:50.236 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8909999999999995e-06 2024-04-26T01:08:50.236 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.236 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.236 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.237 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.497370+0000", 2024-04-26T01:08:50.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5383999999999997e-05 2024-04-26T01:08:50.237 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.237 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.237 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.497409+0000", 2024-04-26T01:08:50.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8705999999999998e-05 2024-04-26T01:08:50.237 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.237 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.237 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.237 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.237 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 81/44 ECSubWriteReply(tid=472, last_complete=81'91, committed=1, applied=1))", 2024-04-26T01:08:50.237 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:47.346604+0000", 2024-04-26T01:08:50.238 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.8229051489999999, 2024-04-26T01:08:50.238 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.149190928, 2024-04-26T01:08:50.238 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.238 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.238 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.238 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.238 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.238 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.238 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.346604+0000", 2024-04-26T01:08:50.238 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.238 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.238 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.238 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.238 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.346604+0000", 2024-04-26T01:08:50.239 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.239 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.239 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.239 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.239 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.346605+0000", 2024-04-26T01:08:50.239 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3200000000000005e-07 2024-04-26T01:08:50.239 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.239 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.239 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.239 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.346607+0000", 2024-04-26T01:08:50.239 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.61e-06 2024-04-26T01:08:50.239 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.239 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.239 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.240 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.494524+0000", 2024-04-26T01:08:50.240 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.14791689399999999 2024-04-26T01:08:50.240 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.240 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.240 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.240 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.494529+0000", 2024-04-26T01:08:50.240 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.074e-06 2024-04-26T01:08:50.240 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.240 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.240 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.240 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.494590+0000", 2024-04-26T01:08:50.240 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0866000000000003e-05 2024-04-26T01:08:50.240 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.240 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.240 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.241 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.495795+0000", 2024-04-26T01:08:50.241 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001204952 2024-04-26T01:08:50.241 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.241 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.241 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.241 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.241 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.241 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 81/44 ECSubWriteReply(tid=486, last_complete=81'93, committed=1, applied=1))", 2024-04-26T01:08:50.241 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:47.522994+0000", 2024-04-26T01:08:50.241 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.6465154659999999, 2024-04-26T01:08:50.241 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.18011165200000001, 2024-04-26T01:08:50.241 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.241 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.241 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.242 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.242 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.242 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.522994+0000", 2024-04-26T01:08:50.242 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.242 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.242 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.242 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.522994+0000", 2024-04-26T01:08:50.242 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.242 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.242 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.242 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.522995+0000", 2024-04-26T01:08:50.243 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1900000000000002e-07 2024-04-26T01:08:50.243 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.243 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.243 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.243 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.522999+0000", 2024-04-26T01:08:50.243 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5940000000000004e-06 2024-04-26T01:08:50.243 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.243 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.243 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.243 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.702792+0000", 2024-04-26T01:08:50.243 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.179793074 2024-04-26T01:08:50.243 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.243 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.243 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.243 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.702808+0000", 2024-04-26T01:08:50.244 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6192999999999999e-05 2024-04-26T01:08:50.244 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.244 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.244 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.244 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.702967+0000", 2024-04-26T01:08:50.244 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015911300000000001 2024-04-26T01:08:50.244 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.244 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.244 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.244 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.703105+0000", 2024-04-26T01:08:50.244 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000137959 2024-04-26T01:08:50.244 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.244 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.244 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.245 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.245 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.245 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 81/44 ECSubWriteReply(tid=487, last_complete=81'108, committed=1, applied=1))", 2024-04-26T01:08:50.245 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:47.524007+0000", 2024-04-26T01:08:50.245 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.6455023440000001, 2024-04-26T01:08:50.245 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.57820720400000003, 2024-04-26T01:08:50.245 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.245 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.245 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.245 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.245 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.245 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.245 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.524007+0000", 2024-04-26T01:08:50.245 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.245 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.246 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.246 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.246 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.524007+0000", 2024-04-26T01:08:50.246 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.246 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.246 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.246 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.246 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.524007+0000", 2024-04-26T01:08:50.246 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7199999999999999e-07 2024-04-26T01:08:50.246 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.246 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.246 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.246 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.524011+0000", 2024-04-26T01:08:50.247 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.743e-06 2024-04-26T01:08:50.247 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.247 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.247 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.247 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.102021+0000", 2024-04-26T01:08:50.247 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.57800951300000003 2024-04-26T01:08:50.247 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.247 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.247 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.247 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.102033+0000", 2024-04-26T01:08:50.247 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2639999999999999e-05 2024-04-26T01:08:50.247 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.247 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.247 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.248 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.102143+0000", 2024-04-26T01:08:50.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000109906 2024-04-26T01:08:50.248 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.248 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.248 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.102214+0000", 2024-04-26T01:08:50.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.093e-05 2024-04-26T01:08:50.248 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.248 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.248 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.248 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.248 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 81/44 ECSubWriteReply(tid=487, last_complete=81'108, committed=1, applied=1))", 2024-04-26T01:08:50.248 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:47.524241+0000", 2024-04-26T01:08:50.248 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.6452677800000002, 2024-04-26T01:08:50.249 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.48392836299999997, 2024-04-26T01:08:50.249 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.249 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.249 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.249 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.249 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.249 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.249 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.524241+0000", 2024-04-26T01:08:50.249 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.249 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.249 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.249 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.249 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.524241+0000", 2024-04-26T01:08:50.249 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.250 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.250 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.524242+0000", 2024-04-26T01:08:50.250 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0999999999999998e-07 2024-04-26T01:08:50.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.250 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.250 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.524246+0000", 2024-04-26T01:08:50.250 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8569999999999997e-06 2024-04-26T01:08:50.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.250 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.251 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.007940+0000", 2024-04-26T01:08:50.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.483694177 2024-04-26T01:08:50.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.251 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.251 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.007954+0000", 2024-04-26T01:08:50.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4260000000000001e-05 2024-04-26T01:08:50.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.251 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.251 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.008108+0000", 2024-04-26T01:08:50.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015377899999999999 2024-04-26T01:08:50.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.252 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.252 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.008170+0000", 2024-04-26T01:08:50.252 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1580000000000003e-05 2024-04-26T01:08:50.252 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.252 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.252 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.252 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.252 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.252 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 83/44 ECSubWriteReply(tid=499, last_complete=83'113, committed=1, applied=1))", 2024-04-26T01:08:50.252 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:49.358147+0000", 2024-04-26T01:08:50.252 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.81136224700000004, 2024-04-26T01:08:50.252 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35853090500000001, 2024-04-26T01:08:50.252 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.252 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.358147+0000", 2024-04-26T01:08:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.358147+0000", 2024-04-26T01:08:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.358148+0000", 2024-04-26T01:08:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.44e-06 2024-04-26T01:08:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.358158+0000", 2024-04-26T01:08:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0869999999999996e-06 2024-04-26T01:08:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.715473+0000", 2024-04-26T01:08:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.35731495000000002 2024-04-26T01:08:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.255 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.255 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.715488+0000", 2024-04-26T01:08:50.255 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5607000000000002e-05 2024-04-26T01:08:50.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.255 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.255 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.255 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.715654+0000", 2024-04-26T01:08:50.255 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016627600000000001 2024-04-26T01:08:50.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.255 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.255 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.255 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.716678+0000", 2024-04-26T01:08:50.255 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0010235450000000001 2024-04-26T01:08:50.255 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.255 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.256 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.256 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.256 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.256 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:50.257 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:08:50.262 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:50.259+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:50.317 INFO:teuthology.orchestra.run.smithi164.stderr:Exporting 3.3s6 info 3.3s6( v 72'20 (0'0,72'20] local-lis/les=30/31 n=2 ec=30/30 lis/c=30/30 les/c/f=31/31/0 sis=30) 2024-04-26T01:08:50.324 INFO:teuthology.orchestra.run.smithi164.stderr:Read 6#3:c992c93e:::smithi16428812-37:15# 2024-04-26T01:08:50.326 INFO:teuthology.orchestra.run.smithi164.stderr:Read 6#3:c992c93e:::smithi16428812-37:head# 2024-04-26T01:08:50.327 INFO:tasks.rados.rados.0.smithi164.stdout:344: done (6 left) 2024-04-26T01:08:50.327 INFO:tasks.rados.rados.0.smithi164.stdout:346: done (5 left) 2024-04-26T01:08:50.327 INFO:tasks.rados.rados.0.smithi164.stdout:354: done (4 left) 2024-04-26T01:08:50.327 INFO:tasks.rados.rados.0.smithi164.stdout:360: done (3 left) 2024-04-26T01:08:50.327 INFO:tasks.rados.rados.0.smithi164.stdout:361: done (2 left) 2024-04-26T01:08:50.327 INFO:tasks.rados.rados.0.smithi164.stdout:362: done (1 left) 2024-04-26T01:08:50.327 INFO:tasks.rados.rados.0.smithi164.stdout:364: done (0 left) 2024-04-26T01:08:50.327 INFO:tasks.rados.rados.0.smithi164.stdout:365: rmattr oid 25 current snap is 28 2024-04-26T01:08:50.327 INFO:tasks.rados.rados.0.smithi164.stdout:365: done (0 left) 2024-04-26T01:08:50.327 INFO:tasks.rados.rados.0.smithi164.stdout:366: snap_remove snap 22 2024-04-26T01:08:50.328 INFO:teuthology.orchestra.run.smithi164.stderr:Export successful 2024-04-26T01:08:50.348 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-26T01:08:50.393 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:50.392+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:50.433 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:08:50.437 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:50.437 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:50.437 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-04-26T01:08:50.437 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:50.454 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:08:50.464 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:50.461+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:50.564 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:50.560+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:50.588 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:50.589 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:50.615 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:50.619 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:50.619 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-04-26T01:08:50.619 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-04-26T01:08:50.620 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-04-26T01:08:50.620 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:50.634 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:08:50.641 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:50.641 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:50.642 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-26T01:08:50.665 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:50.663+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:50.667 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:50.667 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:50.705 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:50.705 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:50.732 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:50.733 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:50.760 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:50.760 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:50.765 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:50.764+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:50.775 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:08:50.776 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-04-26T01:08:50.776 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-04-26T01:08:50.776 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-04-26T01:08:50.776 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.776 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:16 3.4s0 3:2026e263:::smithi16428812-6:head [append 0~98304,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:50.776 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.307585+0000", 2024-04-26T01:08:50.776 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 51.466866719999999, 2024-04-26T01:08:50.776 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.016360486, 2024-04-26T01:08:50.776 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.776 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.776 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:50.777 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.777 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.777 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.777 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 16 2024-04-26T01:08:50.777 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.777 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.777 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.777 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.777 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307585+0000", 2024-04-26T01:08:50.777 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.777 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.777 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.777 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.777 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307585+0000", 2024-04-26T01:08:50.778 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.778 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.778 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.778 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.778 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.307587+0000", 2024-04-26T01:08:50.778 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1280000000000002e-06 2024-04-26T01:08:50.778 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.778 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.778 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.778 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308008+0000", 2024-04-26T01:08:50.778 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00042080399999999999 2024-04-26T01:08:50.778 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.778 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.778 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.779 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308031+0000", 2024-04-26T01:08:50.779 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3085999999999999e-05 2024-04-26T01:08:50.779 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.779 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.779 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.779 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308041+0000", 2024-04-26T01:08:50.779 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9650000000000003e-06 2024-04-26T01:08:50.779 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.779 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.779 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.779 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308110+0000", 2024-04-26T01:08:50.779 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.915e-05 2024-04-26T01:08:50.779 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.779 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.780 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:50.780 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.308277+0000", 2024-04-26T01:08:50.780 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016695800000000001 2024-04-26T01:08:50.780 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.780 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.780 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:50.780 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.309191+0000", 2024-04-26T01:08:50.780 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00091407599999999999 2024-04-26T01:08:50.781 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.781 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.781 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:50.781 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.314046+0000", 2024-04-26T01:08:50.781 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.004854377 2024-04-26T01:08:50.781 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.781 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.781 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-04-26T01:08:50.781 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323807+0000", 2024-04-26T01:08:50.781 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0097610869999999999 2024-04-26T01:08:50.782 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.782 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.782 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.782 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.323946+0000", 2024-04-26T01:08:50.782 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013885500000000001 2024-04-26T01:08:50.782 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.782 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.782 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.782 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.782 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.782 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:18 3.4s0 3:2026e263:::smithi16428812-6:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:50.782 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:07:59.336046+0000", 2024-04-26T01:08:50.782 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 51.438405590000002, 2024-04-26T01:08:50.782 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013179965, 2024-04-26T01:08:50.783 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.783 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.783 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:50.783 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.783 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.783 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.783 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 18 2024-04-26T01:08:50.783 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.783 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.783 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.784 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.784 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336046+0000", 2024-04-26T01:08:50.784 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.784 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.784 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.784 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.784 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336046+0000", 2024-04-26T01:08:50.784 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.784 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.784 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.784 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.784 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336047+0000", 2024-04-26T01:08:50.784 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1119999999999999e-06 2024-04-26T01:08:50.784 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.784 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.785 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.785 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336052+0000", 2024-04-26T01:08:50.785 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3510000000000004e-06 2024-04-26T01:08:50.785 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.785 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.785 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.785 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336063+0000", 2024-04-26T01:08:50.785 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1722999999999999e-05 2024-04-26T01:08:50.785 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.786 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.786 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.786 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.336070+0000", 2024-04-26T01:08:50.786 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6259999999999997e-06 2024-04-26T01:08:50.786 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.786 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.786 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.786 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.338893+0000", 2024-04-26T01:08:50.786 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0028230009999999999 2024-04-26T01:08:50.786 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.786 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.786 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:50.786 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.339060+0000", 2024-04-26T01:08:50.787 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016734700000000001 2024-04-26T01:08:50.787 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.787 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.787 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.787 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.346619+0000", 2024-04-26T01:08:50.787 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0075586949999999998 2024-04-26T01:08:50.787 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.787 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.787 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:50.787 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.346731+0000", 2024-04-26T01:08:50.787 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000111863 2024-04-26T01:08:50.787 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.788 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.788 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.788 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:07:59.349226+0000", 2024-04-26T01:08:50.788 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024952469999999999 2024-04-26T01:08:50.788 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.788 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.788 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.788 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.788 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.788 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.5s1 42/30 ECSubWrite(tid=32, reqid=client.4521.0:351, at_version=42'28, trim_to=0'0, roll_forward_to=42'25))", 2024-04-26T01:08:50.788 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:08.417770+0000", 2024-04-26T01:08:50.788 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 42.356681721999998, 2024-04-26T01:08:50.789 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.64597556300000003, 2024-04-26T01:08:50.789 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.789 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.789 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.789 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.789 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.789 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.789 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.417770+0000", 2024-04-26T01:08:50.789 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.790 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.790 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.790 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.790 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.417770+0000", 2024-04-26T01:08:50.790 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.790 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.790 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.790 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.790 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.417771+0000", 2024-04-26T01:08:50.790 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6499999999999998e-07 2024-04-26T01:08:50.790 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.790 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.791 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.791 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:08.417789+0000", 2024-04-26T01:08:50.791 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8198000000000001e-05 2024-04-26T01:08:50.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.791 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.791 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.791 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.062086+0000", 2024-04-26T01:08:50.791 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.64429725800000004 2024-04-26T01:08:50.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.791 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.791 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.791 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.062104+0000", 2024-04-26T01:08:50.791 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8053000000000001e-05 2024-04-26T01:08:50.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.792 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.792 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.792 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.062287+0000", 2024-04-26T01:08:50.792 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00018218499999999999 2024-04-26T01:08:50.792 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.792 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.792 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:50.792 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.062316+0000", 2024-04-26T01:08:50.792 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9547e-05 2024-04-26T01:08:50.792 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.792 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.792 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:50.792 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.063675+0000", 2024-04-26T01:08:50.792 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001358831 2024-04-26T01:08:50.793 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.793 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.793 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.793 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.063746+0000", 2024-04-26T01:08:50.793 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0625999999999998e-05 2024-04-26T01:08:50.793 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.793 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.793 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.793 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.793 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.793 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(osd.14.21:2 3.cs1 3.83402d7c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:50.793 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.432903+0000", 2024-04-26T01:08:50.793 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 41.341549161000003, 2024-04-26T01:08:50.793 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97310677899999998, 2024-04-26T01:08:50.793 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.794 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.794 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.794 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.794 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.794 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.794 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432903+0000", 2024-04-26T01:08:50.794 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.794 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.794 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.794 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.794 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432903+0000", 2024-04-26T01:08:50.794 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.794 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.794 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.795 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.795 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432904+0000", 2024-04-26T01:08:50.795 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4839999999999999e-06 2024-04-26T01:08:50.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.795 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.795 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432912+0000", 2024-04-26T01:08:50.795 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3489999999999992e-06 2024-04-26T01:08:50.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.795 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.795 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432927+0000", 2024-04-26T01:08:50.795 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4297000000000001e-05 2024-04-26T01:08:50.796 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.796 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432935+0000", 2024-04-26T01:08:50.796 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0360000000000006e-06 2024-04-26T01:08:50.796 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.796 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.432997+0000", 2024-04-26T01:08:50.796 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2176999999999998e-05 2024-04-26T01:08:50.796 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:50.796 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.433000+0000", 2024-04-26T01:08:50.797 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1990000000000002e-06 2024-04-26T01:08:50.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.797 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.797 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.797 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.405993+0000", 2024-04-26T01:08:50.797 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97299321100000002 2024-04-26T01:08:50.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.797 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.797 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.797 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406009+0000", 2024-04-26T01:08:50.797 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6025999999999999e-05 2024-04-26T01:08:50.797 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.797 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.797 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.798 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.798 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:375 3.cs1 3.33e19ddc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:50.798 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.436798+0000", 2024-04-26T01:08:50.798 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 41.337653766000003, 2024-04-26T01:08:50.798 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96902882700000004, 2024-04-26T01:08:50.798 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.798 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.798 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.798 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.798 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.798 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.798 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 375 2024-04-26T01:08:50.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.799 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436798+0000", 2024-04-26T01:08:50.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436798+0000", 2024-04-26T01:08:50.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436800+0000", 2024-04-26T01:08:50.800 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.756e-06 2024-04-26T01:08:50.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.800 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.800 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436810+0000", 2024-04-26T01:08:50.800 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9860000000000006e-06 2024-04-26T01:08:50.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.800 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.800 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436834+0000", 2024-04-26T01:08:50.800 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4457e-05 2024-04-26T01:08:50.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.801 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.801 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436846+0000", 2024-04-26T01:08:50.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1963999999999999e-05 2024-04-26T01:08:50.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.801 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.801 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436916+0000", 2024-04-26T01:08:50.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9667999999999998e-05 2024-04-26T01:08:50.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.801 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:50.801 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436918+0000", 2024-04-26T01:08:50.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.294e-06 2024-04-26T01:08:50.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.405810+0000", 2024-04-26T01:08:50.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96889232599999997 2024-04-26T01:08:50.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.405827+0000", 2024-04-26T01:08:50.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6376e-05 2024-04-26T01:08:50.802 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.802 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.803 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.803 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.803 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.803 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:376 3.cs1 3.33e19ddc (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:50.803 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.436879+0000", 2024-04-26T01:08:50.803 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 41.337572428999998, 2024-04-26T01:08:50.803 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96918375199999995, 2024-04-26T01:08:50.803 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.803 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.803 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.803 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.803 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.804 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.804 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 376 2024-04-26T01:08:50.804 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.804 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.804 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.804 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.804 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436879+0000", 2024-04-26T01:08:50.804 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.804 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.804 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.804 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.804 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436879+0000", 2024-04-26T01:08:50.804 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.804 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.804 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.805 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.805 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436880+0000", 2024-04-26T01:08:50.805 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0100000000000004e-07 2024-04-26T01:08:50.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.805 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.805 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436885+0000", 2024-04-26T01:08:50.805 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3959999999999999e-06 2024-04-26T01:08:50.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.805 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.805 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436896+0000", 2024-04-26T01:08:50.805 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1291000000000001e-05 2024-04-26T01:08:50.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.806 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.806 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436904+0000", 2024-04-26T01:08:50.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6000000000000007e-06 2024-04-26T01:08:50.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.806 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.806 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436935+0000", 2024-04-26T01:08:50.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0626000000000001e-05 2024-04-26T01:08:50.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.806 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:50.806 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.436936+0000", 2024-04-26T01:08:50.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.44e-07 2024-04-26T01:08:50.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.807 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.807 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406050+0000", 2024-04-26T01:08:50.807 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96911480699999997 2024-04-26T01:08:50.807 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.807 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.807 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406063+0000", 2024-04-26T01:08:50.807 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2687000000000001e-05 2024-04-26T01:08:50.807 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.807 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.807 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.807 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.808 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:380 3.cs1 3.f9ba2d2c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:50.808 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.444825+0000", 2024-04-26T01:08:50.808 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 41.329626767000001, 2024-04-26T01:08:50.808 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96128573799999995, 2024-04-26T01:08:50.808 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.808 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.808 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.808 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.808 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.808 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.808 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 380 2024-04-26T01:08:50.808 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.808 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.808 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.809 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.809 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.444825+0000", 2024-04-26T01:08:50.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.809 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.809 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.444825+0000", 2024-04-26T01:08:50.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.809 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.809 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.444827+0000", 2024-04-26T01:08:50.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.731e-06 2024-04-26T01:08:50.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.810 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.810 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.445681+0000", 2024-04-26T01:08:50.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00085380000000000005 2024-04-26T01:08:50.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.810 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.810 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.445706+0000", 2024-04-26T01:08:50.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5768000000000001e-05 2024-04-26T01:08:50.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.810 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.810 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.445720+0000", 2024-04-26T01:08:50.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3788000000000001e-05 2024-04-26T01:08:50.811 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.811 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.811 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.811 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.445801+0000", 2024-04-26T01:08:50.811 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1048000000000001e-05 2024-04-26T01:08:50.811 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.811 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.811 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:50.811 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.445805+0000", 2024-04-26T01:08:50.811 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7479999999999999e-06 2024-04-26T01:08:50.811 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.811 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.811 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.811 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406098+0000", 2024-04-26T01:08:50.812 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96029355199999999 2024-04-26T01:08:50.812 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.812 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.812 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.812 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406111+0000", 2024-04-26T01:08:50.812 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2303000000000001e-05 2024-04-26T01:08:50.812 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.812 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.812 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.812 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.812 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.812 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:381 3.cs1 3.f9ba2d2c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:50.813 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.450315+0000", 2024-04-26T01:08:50.813 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 41.324136451000001, 2024-04-26T01:08:50.813 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95584751300000004, 2024-04-26T01:08:50.813 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.813 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.813 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.813 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.813 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.813 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.813 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 381 2024-04-26T01:08:50.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.813 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.813 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.450315+0000", 2024-04-26T01:08:50.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.450315+0000", 2024-04-26T01:08:50.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.450317+0000", 2024-04-26T01:08:50.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.271e-06 2024-04-26T01:08:50.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451037+0000", 2024-04-26T01:08:50.815 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00071991199999999996 2024-04-26T01:08:50.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451053+0000", 2024-04-26T01:08:50.817 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6049000000000002e-05 2024-04-26T01:08:50.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.817 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.817 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451060+0000", 2024-04-26T01:08:50.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8839999999999994e-06 2024-04-26T01:08:50.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451144+0000", 2024-04-26T01:08:50.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3251999999999995e-05 2024-04-26T01:08:50.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:50.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451147+0000", 2024-04-26T01:08:50.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.994e-06 2024-04-26T01:08:50.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406150+0000", 2024-04-26T01:08:50.819 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95500295999999996 2024-04-26T01:08:50.819 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.819 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406163+0000", 2024-04-26T01:08:50.819 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3191e-05 2024-04-26T01:08:50.819 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.819 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.819 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.819 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.819 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:382 3.cs1 3.f9ba2d2c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:50.819 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.451079+0000", 2024-04-26T01:08:50.819 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 41.323372714000001, 2024-04-26T01:08:50.820 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95513123099999997, 2024-04-26T01:08:50.820 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.820 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.820 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.820 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.820 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.820 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.820 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 382 2024-04-26T01:08:50.820 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.820 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.820 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.820 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451079+0000", 2024-04-26T01:08:50.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.821 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451079+0000", 2024-04-26T01:08:50.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.821 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451080+0000", 2024-04-26T01:08:50.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9599999999999999e-07 2024-04-26T01:08:50.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451116+0000", 2024-04-26T01:08:50.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6797e-05 2024-04-26T01:08:50.822 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451137+0000", 2024-04-26T01:08:50.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0163e-05 2024-04-26T01:08:50.822 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451142+0000", 2024-04-26T01:08:50.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6450000000000002e-06 2024-04-26T01:08:50.823 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.823 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.823 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.823 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451200+0000", 2024-04-26T01:08:50.823 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7703e-05 2024-04-26T01:08:50.823 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.823 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.823 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:50.823 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451201+0000", 2024-04-26T01:08:50.823 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1200000000000001e-07 2024-04-26T01:08:50.824 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.824 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.824 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.824 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406198+0000", 2024-04-26T01:08:50.824 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95499736000000002 2024-04-26T01:08:50.824 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.824 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.824 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.824 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406210+0000", 2024-04-26T01:08:50.824 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2055000000000001e-05 2024-04-26T01:08:50.824 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.824 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.825 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.825 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.825 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.825 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:383 3.cs1 3.f9ba2d2c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:50.825 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.451164+0000", 2024-04-26T01:08:50.825 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 41.323287352000001, 2024-04-26T01:08:50.825 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95510495399999995, 2024-04-26T01:08:50.825 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.825 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.825 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.825 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.826 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.826 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.826 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 383 2024-04-26T01:08:50.826 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.826 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.826 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.826 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.826 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451164+0000", 2024-04-26T01:08:50.826 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.826 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.826 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.826 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.826 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451164+0000", 2024-04-26T01:08:50.827 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.827 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.827 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.827 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.827 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451165+0000", 2024-04-26T01:08:50.827 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.58e-07 2024-04-26T01:08:50.827 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.827 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.827 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.827 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451168+0000", 2024-04-26T01:08:50.827 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3000000000000002e-06 2024-04-26T01:08:50.828 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.828 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.828 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.828 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451174+0000", 2024-04-26T01:08:50.828 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8220000000000003e-06 2024-04-26T01:08:50.828 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.828 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.828 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.828 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451179+0000", 2024-04-26T01:08:50.828 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4700000000000004e-06 2024-04-26T01:08:50.828 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.828 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.829 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.829 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451233+0000", 2024-04-26T01:08:50.829 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4086999999999999e-05 2024-04-26T01:08:50.829 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.829 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.829 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:50.829 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451234+0000", 2024-04-26T01:08:50.829 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0580000000000001e-06 2024-04-26T01:08:50.829 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.829 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.829 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.830 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406253+0000", 2024-04-26T01:08:50.830 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95501902800000005 2024-04-26T01:08:50.830 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.830 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.830 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.830 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406269+0000", 2024-04-26T01:08:50.830 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6731e-05 2024-04-26T01:08:50.830 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.831 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.831 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.831 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.831 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.831 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:384 3.cs1 3.f9ba2d2c (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:50.831 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:09.451185+0000", 2024-04-26T01:08:50.831 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 41.323266678000003, 2024-04-26T01:08:50.831 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95528391300000004, 2024-04-26T01:08:50.831 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.831 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.831 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.831 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.831 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.832 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.832 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 384 2024-04-26T01:08:50.832 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.832 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.832 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.832 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.832 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451185+0000", 2024-04-26T01:08:50.832 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.832 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.832 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.832 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.832 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451185+0000", 2024-04-26T01:08:50.832 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451186+0000", 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3599999999999999e-07 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451187+0000", 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.804e-06 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451193+0000", 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4060000000000004e-06 2024-04-26T01:08:50.833 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.834 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.834 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.834 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451197+0000", 2024-04-26T01:08:50.834 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.262e-06 2024-04-26T01:08:50.834 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.834 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.834 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.834 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451254+0000", 2024-04-26T01:08:50.834 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6635000000000002e-05 2024-04-26T01:08:50.834 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.834 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.834 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:50.834 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:09.451254+0000", 2024-04-26T01:08:50.835 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6599999999999995e-07 2024-04-26T01:08:50.835 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.835 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.835 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.835 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406454+0000", 2024-04-26T01:08:50.835 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95519958699999996 2024-04-26T01:08:50.835 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.835 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.835 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.835 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:10.406469+0000", 2024-04-26T01:08:50.835 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5017e-05 2024-04-26T01:08:50.835 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.835 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.835 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.835 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.836 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.836 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:483 3.4s0 3:2d34bde3:::smithi16428812-29:head [copy-from ver 0] snapc a=[a,8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-26T01:08:50.836 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:14.042718+0000", 2024-04-26T01:08:50.836 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 36.731734107000001, 2024-04-26T01:08:50.836 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015567493, 2024-04-26T01:08:50.836 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.836 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.836 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:50.836 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.836 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.836 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.837 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 483 2024-04-26T01:08:50.837 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.837 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.837 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.837 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.837 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042718+0000", 2024-04-26T01:08:50.837 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.837 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.837 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.837 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.837 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042718+0000", 2024-04-26T01:08:50.837 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.837 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.837 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.837 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.838 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042721+0000", 2024-04-26T01:08:50.838 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.2100000000000002e-06 2024-04-26T01:08:50.838 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.838 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.838 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.838 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042756+0000", 2024-04-26T01:08:50.838 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4972000000000003e-05 2024-04-26T01:08:50.838 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.838 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.838 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.838 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042793+0000", 2024-04-26T01:08:50.838 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7027000000000003e-05 2024-04-26T01:08:50.838 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.838 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.839 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.839 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042819+0000", 2024-04-26T01:08:50.839 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6359999999999998e-05 2024-04-26T01:08:50.839 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.839 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.839 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.839 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042991+0000", 2024-04-26T01:08:50.839 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017123400000000001 2024-04-26T01:08:50.839 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.839 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.839 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op must wait for map", 2024-04-26T01:08:50.840 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.043005+0000", 2024-04-26T01:08:50.840 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4414e-05 2024-04-26T01:08:50.840 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.840 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.840 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.840 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.047524+0000", 2024-04-26T01:08:50.840 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0045195349999999999 2024-04-26T01:08:50.840 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.840 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.840 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:50.840 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.047801+0000", 2024-04-26T01:08:50.840 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00027666799999999999 2024-04-26T01:08:50.841 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.841 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.841 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.841 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.058285+0000", 2024-04-26T01:08:50.841 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010484073 2024-04-26T01:08:50.841 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.841 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.841 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.841 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.841 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.841 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:484 3.4s0 3:2d34bde3:::smithi16428812-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-26T01:08:50.841 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:14.042874+0000", 2024-04-26T01:08:50.842 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 36.731578042999999, 2024-04-26T01:08:50.842 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015940147000000002, 2024-04-26T01:08:50.842 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.842 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.842 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:50.842 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.842 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.842 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.842 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 484 2024-04-26T01:08:50.842 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.842 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.843 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.843 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.843 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042874+0000", 2024-04-26T01:08:50.843 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.843 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.843 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.843 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.843 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042874+0000", 2024-04-26T01:08:50.843 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.843 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.843 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.843 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.843 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042875+0000", 2024-04-26T01:08:50.843 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.64e-06 2024-04-26T01:08:50.844 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.844 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.844 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.844 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042884+0000", 2024-04-26T01:08:50.844 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1100000000000003e-06 2024-04-26T01:08:50.844 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.844 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.844 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.844 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042909+0000", 2024-04-26T01:08:50.844 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5857e-05 2024-04-26T01:08:50.844 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.844 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.845 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.845 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.042929+0000", 2024-04-26T01:08:50.845 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9167000000000001e-05 2024-04-26T01:08:50.845 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.845 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.845 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.845 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.043135+0000", 2024-04-26T01:08:50.845 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00020666799999999999 2024-04-26T01:08:50.845 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.845 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.845 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:50.845 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.043158+0000", 2024-04-26T01:08:50.845 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.304e-05 2024-04-26T01:08:50.845 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.846 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.846 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.846 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.048261+0000", 2024-04-26T01:08:50.846 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0051031039999999998 2024-04-26T01:08:50.846 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.846 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.846 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for blocked object", 2024-04-26T01:08:50.846 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.048407+0000", 2024-04-26T01:08:50.846 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000145308 2024-04-26T01:08:50.846 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.846 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.846 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.846 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.058472+0000", 2024-04-26T01:08:50.847 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010065703 2024-04-26T01:08:50.847 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.847 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.847 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:50.847 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.058641+0000", 2024-04-26T01:08:50.847 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016832199999999999 2024-04-26T01:08:50.847 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.847 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.847 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.847 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:14.058814+0000", 2024-04-26T01:08:50.847 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017322799999999999 2024-04-26T01:08:50.847 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.847 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.847 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.847 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.848 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.848 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:606 3.4s0 3:2d34bde3:::smithi16428812-29:head [read 0~2129920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:50.848 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.340889+0000", 2024-04-26T01:08:50.848 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 26.433562549000001, 2024-04-26T01:08:50.848 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012634348, 2024-04-26T01:08:50.848 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.848 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.848 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:50.848 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.848 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.848 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.848 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 606 2024-04-26T01:08:50.848 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.848 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340889+0000", 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340889+0000", 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340890+0000", 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6299999999999993e-07 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.849 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.850 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.850 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340896+0000", 2024-04-26T01:08:50.850 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6500000000000001e-06 2024-04-26T01:08:50.850 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.850 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.850 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.850 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340917+0000", 2024-04-26T01:08:50.850 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0769999999999999e-05 2024-04-26T01:08:50.850 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.850 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.850 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.850 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.340927+0000", 2024-04-26T01:08:50.850 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0077e-05 2024-04-26T01:08:50.850 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.851 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.851 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.851 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341566+0000", 2024-04-26T01:08:50.851 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000638881 2024-04-26T01:08:50.851 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.851 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.851 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:50.851 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341677+0000", 2024-04-26T01:08:50.851 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000111595 2024-04-26T01:08:50.851 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.851 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.851 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.851 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.353524+0000", 2024-04-26T01:08:50.851 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011846412000000001 2024-04-26T01:08:50.851 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.852 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.852 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.852 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.852 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.852 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:607 3.4s0 3:2d34bde3:::smithi16428812-29:head [read 0~2129920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:08:50.852 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:24.341454+0000", 2024-04-26T01:08:50.852 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 26.432998064, 2024-04-26T01:08:50.852 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014935789, 2024-04-26T01:08:50.852 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.852 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.852 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:50.852 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.852 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.852 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.853 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 607 2024-04-26T01:08:50.853 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.853 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.853 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.853 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.853 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341454+0000", 2024-04-26T01:08:50.853 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.853 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.853 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.853 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.854 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341454+0000", 2024-04-26T01:08:50.854 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.854 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.854 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.854 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.854 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341455+0000", 2024-04-26T01:08:50.854 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0550000000000001e-06 2024-04-26T01:08:50.854 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.854 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.854 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.854 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341464+0000", 2024-04-26T01:08:50.854 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4350000000000003e-06 2024-04-26T01:08:50.854 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.855 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.855 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.855 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341477+0000", 2024-04-26T01:08:50.855 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2731999999999999e-05 2024-04-26T01:08:50.855 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.855 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.855 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.855 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.341488+0000", 2024-04-26T01:08:50.855 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.118e-05 2024-04-26T01:08:50.855 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.856 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.856 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.856 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.343260+0000", 2024-04-26T01:08:50.856 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0017719980000000001 2024-04-26T01:08:50.856 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.856 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.856 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:50.856 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.343498+0000", 2024-04-26T01:08:50.856 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00023756600000000001 2024-04-26T01:08:50.856 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.856 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.856 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.856 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:24.356390+0000", 2024-04-26T01:08:50.856 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012891823 2024-04-26T01:08:50.856 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.857 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.857 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.857 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.857 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.857 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 59/30 ECSubWriteReply(tid=56, last_complete=59'40, committed=1, applied=1))", 2024-04-26T01:08:50.857 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:25.412873+0000", 2024-04-26T01:08:50.857 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 25.361578798, 2024-04-26T01:08:50.857 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.43333044199999998, 2024-04-26T01:08:50.857 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.857 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.412873+0000", 2024-04-26T01:08:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.412873+0000", 2024-04-26T01:08:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.412874+0000", 2024-04-26T01:08:50.859 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3600000000000003e-07 2024-04-26T01:08:50.859 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.859 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.859 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.859 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.412881+0000", 2024-04-26T01:08:50.859 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3699999999999997e-06 2024-04-26T01:08:50.859 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.859 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.859 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.859 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.846041+0000", 2024-04-26T01:08:50.860 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.43316028400000001 2024-04-26T01:08:50.860 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.860 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.860 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.860 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.846075+0000", 2024-04-26T01:08:50.860 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3213000000000002e-05 2024-04-26T01:08:50.860 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.860 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.860 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.860 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.846162+0000", 2024-04-26T01:08:50.860 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7017999999999999e-05 2024-04-26T01:08:50.860 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.860 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.860 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.860 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:25.846203+0000", 2024-04-26T01:08:50.861 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1921000000000001e-05 2024-04-26T01:08:50.861 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.861 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.861 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.861 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.861 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.861 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:879 3.as0 3:592ac288:::smithi16428812-20:head [read 0~1490944,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:08:50.861 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:47.309487+0000", 2024-04-26T01:08:50.861 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.4649652849999999, 2024-04-26T01:08:50.861 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.040349178, 2024-04-26T01:08:50.861 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.862 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.862 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:50.862 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.862 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.862 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.862 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 879 2024-04-26T01:08:50.862 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.862 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.862 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.862 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.862 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309487+0000", 2024-04-26T01:08:50.862 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.863 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.863 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.863 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.863 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309487+0000", 2024-04-26T01:08:50.863 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.863 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.863 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.863 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.863 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309490+0000", 2024-04-26T01:08:50.863 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8240000000000001e-06 2024-04-26T01:08:50.863 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.863 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.864 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.864 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309507+0000", 2024-04-26T01:08:50.864 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6594000000000001e-05 2024-04-26T01:08:50.864 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.864 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.864 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.864 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309548+0000", 2024-04-26T01:08:50.864 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1017999999999999e-05 2024-04-26T01:08:50.864 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.864 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.864 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.864 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309569+0000", 2024-04-26T01:08:50.865 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1515999999999999e-05 2024-04-26T01:08:50.865 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.865 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.865 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.865 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.315466+0000", 2024-04-26T01:08:50.865 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0058964430000000003 2024-04-26T01:08:50.865 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.865 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.865 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:50.865 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.315471+0000", 2024-04-26T01:08:50.865 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9599999999999999e-06 2024-04-26T01:08:50.865 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.865 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.865 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.866 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.329957+0000", 2024-04-26T01:08:50.866 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0144864600000001 2024-04-26T01:08:50.866 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.866 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.866 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for missing object", 2024-04-26T01:08:50.866 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.330752+0000", 2024-04-26T01:08:50.866 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00079475699999999999 2024-04-26T01:08:50.866 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.866 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.866 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.866 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.342768+0000", 2024-04-26T01:08:50.866 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012015981 2024-04-26T01:08:50.866 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.866 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.867 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:50.867 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.342986+0000", 2024-04-26T01:08:50.867 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00021758899999999999 2024-04-26T01:08:50.867 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.867 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.867 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.867 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.349836+0000", 2024-04-26T01:08:50.867 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0068500360000000003 2024-04-26T01:08:50.867 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.867 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.867 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.867 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.867 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.867 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:880 3.as0 3:592ac288:::smithi16428812-20:head [read 0~1490944,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:08:50.867 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:47.309607+0000", 2024-04-26T01:08:50.868 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.4648451590000002, 2024-04-26T01:08:50.868 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0422197609999999, 2024-04-26T01:08:50.868 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.868 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.868 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:50.868 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.868 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.868 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.868 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 880 2024-04-26T01:08:50.868 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.868 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.868 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.868 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.868 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309607+0000", 2024-04-26T01:08:50.869 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.869 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.869 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.869 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.869 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309607+0000", 2024-04-26T01:08:50.869 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.869 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.869 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.869 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.869 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309608+0000", 2024-04-26T01:08:50.869 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.136e-06 2024-04-26T01:08:50.869 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.869 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.869 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.870 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309618+0000", 2024-04-26T01:08:50.870 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0134e-05 2024-04-26T01:08:50.870 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.870 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.870 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.870 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309633+0000", 2024-04-26T01:08:50.870 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.519e-05 2024-04-26T01:08:50.870 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.870 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.870 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.871 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309644+0000", 2024-04-26T01:08:50.871 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0657e-05 2024-04-26T01:08:50.871 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.871 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.871 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.871 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.321269+0000", 2024-04-26T01:08:50.871 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011625527 2024-04-26T01:08:50.871 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.871 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.871 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:50.871 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.321277+0000", 2024-04-26T01:08:50.871 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5839999999999997e-06 2024-04-26T01:08:50.871 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.871 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.872 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.872 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.332278+0000", 2024-04-26T01:08:50.872 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.011001276 2024-04-26T01:08:50.872 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.872 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.872 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for missing object", 2024-04-26T01:08:50.872 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.332478+0000", 2024-04-26T01:08:50.872 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019960099999999999 2024-04-26T01:08:50.872 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.872 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.872 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.873 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.343594+0000", 2024-04-26T01:08:50.873 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011116312 2024-04-26T01:08:50.873 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.873 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.873 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:50.873 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.343738+0000", 2024-04-26T01:08:50.873 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000143523 2024-04-26T01:08:50.873 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.873 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.873 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.873 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.351826+0000", 2024-04-26T01:08:50.873 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0080888209999999995 2024-04-26T01:08:50.873 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.874 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.874 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.874 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.874 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.874 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.4521.0:881 3.as0 3:592ac288:::smithi16428812-20:head [read 0~1490944,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:08:50.874 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:47.309684+0000", 2024-04-26T01:08:50.874 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.4647673960000001, 2024-04-26T01:08:50.874 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.044916537, 2024-04-26T01:08:50.874 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.874 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.874 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-04-26T01:08:50.874 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-04-26T01:08:50.874 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.4521", 2024-04-26T01:08:50.874 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:50.875 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 881 2024-04-26T01:08:50.875 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.875 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.875 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.875 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.875 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309684+0000", 2024-04-26T01:08:50.875 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.875 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.875 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.875 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.875 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309684+0000", 2024-04-26T01:08:50.875 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.875 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.876 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.876 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.876 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309685+0000", 2024-04-26T01:08:50.876 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8999999999999995e-07 2024-04-26T01:08:50.876 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.876 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.876 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.876 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309704+0000", 2024-04-26T01:08:50.876 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9182e-05 2024-04-26T01:08:50.876 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.876 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.876 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.876 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309722+0000", 2024-04-26T01:08:50.876 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7912000000000001e-05 2024-04-26T01:08:50.877 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.877 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.877 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.877 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.309736+0000", 2024-04-26T01:08:50.877 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3407999999999999e-05 2024-04-26T01:08:50.877 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.877 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.877 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.877 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.321354+0000", 2024-04-26T01:08:50.877 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011618102 2024-04-26T01:08:50.877 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.877 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.877 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-04-26T01:08:50.877 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:47.321356+0000", 2024-04-26T01:08:50.878 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8560000000000001e-06 2024-04-26T01:08:50.878 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.878 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.878 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.878 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.332971+0000", 2024-04-26T01:08:50.878 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.011615328 2024-04-26T01:08:50.878 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.878 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.878 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for missing object", 2024-04-26T01:08:50.878 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.333621+0000", 2024-04-26T01:08:50.878 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00064975999999999999 2024-04-26T01:08:50.878 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.878 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.878 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.878 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.347028+0000", 2024-04-26T01:08:50.879 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013406698 2024-04-26T01:08:50.879 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.879 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.879 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-04-26T01:08:50.879 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.347205+0000", 2024-04-26T01:08:50.879 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017724599999999999 2024-04-26T01:08:50.879 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.879 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.879 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.879 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:48.354601+0000", 2024-04-26T01:08:50.879 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0073961549999999997 2024-04-26T01:08:50.879 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.879 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.880 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.880 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.880 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.880 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDECSubOpWrite(3.5s1 83/30 ECSubWrite(tid=71, reqid=osd.15.0:71, at_version=83'58, trim_to=0'0, roll_forward_to=81'56))", 2024-04-26T01:08:50.880 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-04-26T01:08:49.328595+0000", 2024-04-26T01:08:50.880 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.4458563529999999, 2024-04-26T01:08:50.880 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011955001, 2024-04-26T01:08:50.880 INFO:teuthology.orchestra.run.smithi008.stdout: "continuous": false, 2024-04-26T01:08:50.880 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-04-26T01:08:50.880 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-04-26T01:08:50.880 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-04-26T01:08:50.880 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.880 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-04-26T01:08:50.880 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.328595+0000", 2024-04-26T01:08:50.881 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.881 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.881 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.881 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-04-26T01:08:50.881 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.328595+0000", 2024-04-26T01:08:50.881 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-04-26T01:08:50.881 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.881 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.881 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-04-26T01:08:50.881 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.328596+0000", 2024-04-26T01:08:50.881 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1299999999999999e-07 2024-04-26T01:08:50.881 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.881 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.881 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-04-26T01:08:50.882 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.328647+0000", 2024-04-26T01:08:50.882 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1072999999999997e-05 2024-04-26T01:08:50.882 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.882 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.882 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-04-26T01:08:50.882 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.328737+0000", 2024-04-26T01:08:50.882 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0093000000000006e-05 2024-04-26T01:08:50.882 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.882 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.882 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-04-26T01:08:50.882 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.328760+0000", 2024-04-26T01:08:50.882 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.302e-05 2024-04-26T01:08:50.882 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.882 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.882 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-04-26T01:08:50.883 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.331789+0000", 2024-04-26T01:08:50.883 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0030281819999999999 2024-04-26T01:08:50.883 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.883 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.883 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_started", 2024-04-26T01:08:50.883 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.331820+0000", 2024-04-26T01:08:50.883 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1616999999999999e-05 2024-04-26T01:08:50.883 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.883 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.883 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_committed", 2024-04-26T01:08:50.883 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.340444+0000", 2024-04-26T01:08:50.883 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0086235200000000008 2024-04-26T01:08:50.883 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-26T01:08:50.883 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-26T01:08:50.883 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-04-26T01:08:50.884 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-04-26T01:08:49.340550+0000", 2024-04-26T01:08:50.884 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000106783 2024-04-26T01:08:50.884 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.884 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.884 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.884 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-26T01:08:50.884 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-26T01:08:50.884 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:08:50.886 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 3.3s6 2024-04-26T01:08:50.887 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:08:50.887 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:50.887 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:50.888 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:50.888 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:50.888 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:50.888 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:50.889 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:08:50.889 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:08:50.890 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:50.884+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:50.899 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:50.900 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:50.930 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:50.930 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:50.961 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:50.962 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:50.967 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:50.964+0000 7fa1dffff640 0 -- 172.21.15.8:0/2305822261 >> [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] conn(0x7fa1e02237f0 msgr2=0x7fa1e0225be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-04-26T01:08:50.997 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:50.997 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:51.001 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T01:08:51.002 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T01:08:51.020 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:51.020 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:51.034 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:08:51.035 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:50.987+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:51.043 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-04-26T01:08:51.046 INFO:tasks.rados.rados.0.smithi164.stdout:366: done (0 left) 2024-04-26T01:08:51.046 INFO:tasks.rados.rados.0.smithi164.stdout:367: read oid 9 snap -1 2024-04-26T01:08:51.046 INFO:tasks.rados.rados.0.smithi164.stdout:367: expect (ObjNum 99 snap 11 seq_num 99) 2024-04-26T01:08:51.046 INFO:tasks.rados.rados.0.smithi164.stdout:368: read oid 2 snap -1 2024-04-26T01:08:51.046 INFO:tasks.rados.rados.0.smithi164.stdout:368: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-26T01:08:51.046 INFO:tasks.rados.rados.0.smithi164.stdout:369: read oid 47 snap -1 2024-04-26T01:08:51.046 INFO:tasks.rados.rados.0.smithi164.stdout:369: expect (ObjNum 65 snap 3 seq_num 65) 2024-04-26T01:08:51.046 INFO:tasks.rados.rados.0.smithi164.stdout:370: snap_remove snap 26 2024-04-26T01:08:51.087 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:51.087+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:51.174 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:51.175 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:51.189 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:08:51.189 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:51.187+0000 7ff2871de640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 32810) UID: 0 2024-04-26T01:08:51.289 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:51.287+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:51.312 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:51.312 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:51.327 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:08:51.390 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:51.387+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:51.463 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:51.463 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:51.477 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:08:51.491 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:51.488+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:51.592 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:51.591+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:51.606 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:51.607 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:51.621 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:08:51.693 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:51.691+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:51.740 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:51.740 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:51.755 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:08:51.793 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:51.791+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:51.885 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:51.885 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:51.894 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:51.892+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:51.899 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:08:51.995 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:51.993+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:52.020 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:52.020 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:52.035 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:08:52.051 INFO:tasks.rados.rados.0.smithi164.stdout:370: done (3 left) 2024-04-26T01:08:52.051 INFO:tasks.rados.rados.0.smithi164.stdout:371: read oid 28 snap -1 2024-04-26T01:08:52.052 INFO:tasks.rados.rados.0.smithi164.stdout:371: expect deleted 2024-04-26T01:08:52.052 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 25 current snap is 28 2024-04-26T01:08:52.052 INFO:tasks.rados.rados.0.smithi164.stdout:372: seq_num 151 ranges {2326528=131072} 2024-04-26T01:08:52.069 INFO:teuthology.orchestra.run.smithi164.stdout: marking collection for removal 2024-04-26T01:08:52.069 INFO:teuthology.orchestra.run.smithi164.stdout:setting '_remove' omap key 2024-04-26T01:08:52.069 INFO:teuthology.orchestra.run.smithi164.stdout:finish_remove_pgs 3.3s6_head removing 3.3s6 2024-04-26T01:08:52.076 INFO:teuthology.orchestra.run.smithi164.stdout:Remove successful 2024-04-26T01:08:52.080 INFO:tasks.rados.rados.0.smithi164.stdout:372: writing smithi16428812-25 from 2326528 to 2457600 tid 1 2024-04-26T01:08:52.080 INFO:tasks.rados.rados.0.smithi164.stdout:373: rollback oid 13 current snap is 28 2024-04-26T01:08:52.080 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 13 to 25 2024-04-26T01:08:52.080 INFO:tasks.rados.rados.0.smithi164.stdout:368: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-26T01:08:52.081 INFO:tasks.rados.rados.0.smithi164.stdout:367: expect (ObjNum 99 snap 11 seq_num 99) 2024-04-26T01:08:52.123 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:52.121+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:52.133 INFO:tasks.rados.rados.0.smithi164.stdout:369: expect (ObjNum 65 snap 3 seq_num 65) 2024-04-26T01:08:52.197 INFO:tasks.rados.rados.0.smithi164.stdout:372: finishing write tid 1 to smithi16428812-25 2024-04-26T01:08:52.197 INFO:tasks.rados.rados.0.smithi164.stdout:372: finishing write tid 2 to smithi16428812-25 2024-04-26T01:08:52.197 INFO:tasks.rados.rados.0.smithi164.stdout:373: finishing rollback tid 0 to smithi16428812-13 2024-04-26T01:08:52.197 INFO:tasks.rados.rados.0.smithi164.stdout:373: finishing rollback tid 1 to smithi16428812-13 2024-04-26T01:08:52.197 INFO:tasks.rados.rados.0.smithi164.stdout:372: finishing write tid 3 to smithi16428812-25 2024-04-26T01:08:52.197 INFO:tasks.rados.rados.0.smithi164.stdout:372: oid 25 updating version 0 to 112 2024-04-26T01:08:52.197 INFO:tasks.rados.rados.0.smithi164.stdout:372: oid 25 version 112 is already newer than 111 2024-04-26T01:08:52.197 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 25 v 112 (ObjNum 151 snap 28 seq_num 151) dirty exists 2024-04-26T01:08:52.198 INFO:tasks.rados.rados.0.smithi164.stdout:372: left oid 25 (ObjNum 151 snap 28 seq_num 151) 2024-04-26T01:08:52.198 INFO:tasks.rados.rados.0.smithi164.stdout:373: finishing rollback tid 2 to smithi16428812-13 2024-04-26T01:08:52.198 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 13 v 97 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-26T01:08:52.198 INFO:tasks.rados.rados.0.smithi164.stdout:367: done (5 left) 2024-04-26T01:08:52.198 INFO:tasks.rados.rados.0.smithi164.stdout:368: done (4 left) 2024-04-26T01:08:52.198 INFO:tasks.rados.rados.0.smithi164.stdout:369: done (3 left) 2024-04-26T01:08:52.198 INFO:tasks.rados.rados.0.smithi164.stdout:371: done (2 left) 2024-04-26T01:08:52.198 INFO:tasks.rados.rados.0.smithi164.stdout:372: done (1 left) 2024-04-26T01:08:52.199 INFO:tasks.rados.rados.0.smithi164.stdout:373: done (0 left) 2024-04-26T01:08:52.199 INFO:tasks.rados.rados.0.smithi164.stdout:374: read oid 13 snap -1 2024-04-26T01:08:52.199 INFO:tasks.rados.rados.0.smithi164.stdout:374: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-26T01:08:52.199 INFO:tasks.rados.rados.0.smithi164.stdout:375: snap_create 2024-04-26T01:08:52.199 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:52.195+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:52.204 INFO:tasks.rados.rados.0.smithi164.stdout:374: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-26T01:08:52.207 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:52.207 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:52.222 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:08:52.298 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:52.296+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:52.390 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:52.390 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:52.399 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:52.399+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:52.405 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:08:52.500 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:52.497+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:52.531 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:52.531 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:52.546 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:08:52.601 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:52.599+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:52.606 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.3s6.3 2024-04-26T01:08:52.703 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:52.699+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:52.717 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:52.717 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:52.731 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:08:52.803 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:52.803+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:52.857 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:52.857 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:52.872 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:08:52.904 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:52.903+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:52.999 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:53.000 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:53.003 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:08:53.004 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:53.002+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:53.013 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:08:53.052 INFO:tasks.rados.rados.0.smithi164.stdout:374: done (1 left) 2024-04-26T01:08:53.052 INFO:tasks.rados.rados.0.smithi164.stdout:375: done (0 left) 2024-04-26T01:08:53.052 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 32 current snap is 29 2024-04-26T01:08:53.052 INFO:tasks.rados.rados.0.smithi164.stdout:376: seq_num 152 ranges {4161536=147456} 2024-04-26T01:08:53.074 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-26T01:08:53.106 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:53.101+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:53.117 INFO:tasks.rados.rados.0.smithi164.stdout:376: writing smithi16428812-32 from 4161536 to 4308992 tid 1 2024-04-26T01:08:53.117 INFO:tasks.rados.rados.0.smithi164.stdout:377: snap_remove snap 28 2024-04-26T01:08:53.137 INFO:teuthology.orchestra.run.smithi012.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:53.137 INFO:teuthology.orchestra.run.smithi012.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:53.152 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:08:53.175 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:53.175 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:53.199 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:53.200 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:53.206 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:53.201+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:53.226 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:53.226 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:53.226 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-26T01:08:53.264 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:53.264 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:53.282 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:53.282 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:53.283 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:08:53.283 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:53.297 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:08:53.303 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:53.303 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:53.307 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:53.301+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:53.333 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:53.333 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:53.358 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:53.358 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:53.382 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:53.382 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:53.407 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:53.405+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:53.410 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:53.410 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:53.416 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:53.416 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:53.416 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:08:53.416 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:08:53.416 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:53.431 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:08:53.438 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:53.438 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:53.468 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:08:53.468 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:08:53.498 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:53.499 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:53.508 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:53.506+0000 7ff2871de640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 32810) UID: 0 2024-04-26T01:08:53.532 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:53.532 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:53.556 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:53.556 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:08:53.556 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:08:53.556 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:08:53.556 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.557 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:145 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'103, mlcod=25'98)", 2024-04-26T01:08:53.557 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:47.983085+0000", 2024-04-26T01:08:53.557 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 65.573195882999997, 2024-04-26T01:08:53.557 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.055794923000000003, 2024-04-26T01:08:53.557 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.557 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.557 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:53.557 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.557 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.557 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.557 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983085+0000", 2024-04-26T01:08:53.557 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.557 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.557 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.558 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.558 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983085+0000", 2024-04-26T01:08:53.558 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.558 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.558 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983086+0000", 2024-04-26T01:08:53.558 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9699999999999994e-07 2024-04-26T01:08:53.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.558 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.558 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983194+0000", 2024-04-26T01:08:53.559 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107569 2024-04-26T01:08:53.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.559 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.559 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.559 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037255+0000", 2024-04-26T01:08:53.559 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.054061302999999998 2024-04-26T01:08:53.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.559 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.559 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.559 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037268+0000", 2024-04-26T01:08:53.559 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3477e-05 2024-04-26T01:08:53.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.559 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.560 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.560 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037378+0000", 2024-04-26T01:08:53.560 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000110089 2024-04-26T01:08:53.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.560 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.560 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:53.560 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037414+0000", 2024-04-26T01:08:53.560 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5123000000000002e-05 2024-04-26T01:08:53.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.560 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.560 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:53.560 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.038790+0000", 2024-04-26T01:08:53.560 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0013768820000000001 2024-04-26T01:08:53.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.561 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.561 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.561 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.038880+0000", 2024-04-26T01:08:53.561 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9482999999999997e-05 2024-04-26T01:08:53.561 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.561 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.561 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.561 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.561 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpRead(3.2s1 33/30 ECSubRead(tid=163, to_read={3:410d186f:::smithi16428812-11:head=0,4096,0}, subchunks={3:410d186f:::smithi16428812-11:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:53.561 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.340790+0000", 2024-04-26T01:08:53.561 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 54.215490885999998, 2024-04-26T01:08:53.561 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.047962901000000002, 2024-04-26T01:08:53.562 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.562 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.562 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:53.562 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.562 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.562 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.562 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340790+0000", 2024-04-26T01:08:53.562 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.562 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.562 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.562 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340790+0000", 2024-04-26T01:08:53.562 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.563 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.563 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340791+0000", 2024-04-26T01:08:53.563 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.3300000000000001e-07 2024-04-26T01:08:53.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.563 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.563 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340801+0000", 2024-04-26T01:08:53.563 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0308e-05 2024-04-26T01:08:53.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.563 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.563 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388316+0000", 2024-04-26T01:08:53.563 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.047514702999999998 2024-04-26T01:08:53.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.564 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.564 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.564 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388324+0000", 2024-04-26T01:08:53.564 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.3529999999999995e-06 2024-04-26T01:08:53.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.564 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.564 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.564 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388439+0000", 2024-04-26T01:08:53.564 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000114657 2024-04-26T01:08:53.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.564 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.564 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.564 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388753+0000", 2024-04-26T01:08:53.565 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00031404700000000002 2024-04-26T01:08:53.565 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.565 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.565 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.565 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.565 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.565 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:62 3.es0 3:7eef64f7:::smithi16428812-21:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:53.565 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.361875+0000", 2024-04-26T01:08:53.565 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 54.194405811999999, 2024-04-26T01:08:53.565 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.111184554, 2024-04-26T01:08:53.565 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.565 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.565 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:53.565 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:53.566 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:53.566 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:53.566 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 62 2024-04-26T01:08:53.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.566 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.566 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.566 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.566 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361875+0000", 2024-04-26T01:08:53.566 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.566 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.566 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.566 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361875+0000", 2024-04-26T01:08:53.566 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.567 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.567 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361876+0000", 2024-04-26T01:08:53.567 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5899999999999995e-07 2024-04-26T01:08:53.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.567 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.567 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361880+0000", 2024-04-26T01:08:53.567 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8739999999999997e-06 2024-04-26T01:08:53.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.567 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.567 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361887+0000", 2024-04-26T01:08:53.568 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4839999999999998e-06 2024-04-26T01:08:53.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.568 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.568 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361892+0000", 2024-04-26T01:08:53.568 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.437e-06 2024-04-26T01:08:53.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.568 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.568 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.363374+0000", 2024-04-26T01:08:53.568 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001482693 2024-04-26T01:08:53.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.568 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:53.569 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.363442+0000", 2024-04-26T01:08:53.569 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7448999999999997e-05 2024-04-26T01:08:53.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.569 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:53.569 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.363805+0000", 2024-04-26T01:08:53.569 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00036289900000000002 2024-04-26T01:08:53.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.569 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:53.569 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.366199+0000", 2024-04-26T01:08:53.569 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.002394654 2024-04-26T01:08:53.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.570 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:53.570 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472968+0000", 2024-04-26T01:08:53.570 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.106769087 2024-04-26T01:08:53.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.570 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.570 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473060+0000", 2024-04-26T01:08:53.570 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.1218000000000007e-05 2024-04-26T01:08:53.570 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.570 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.570 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.571 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:63 3.es0 3:7eef64f7:::smithi16428812-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:53.571 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.361913+0000", 2024-04-26T01:08:53.571 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 54.194367868, 2024-04-26T01:08:53.571 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.11409399200000001, 2024-04-26T01:08:53.571 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.571 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.571 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:53.571 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:53.571 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:53.571 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:53.571 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 63 2024-04-26T01:08:53.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.571 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.572 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.572 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361913+0000", 2024-04-26T01:08:53.572 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.572 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.572 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361913+0000", 2024-04-26T01:08:53.572 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.572 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.572 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361914+0000", 2024-04-26T01:08:53.572 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.5700000000000002e-07 2024-04-26T01:08:53.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.573 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.573 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361917+0000", 2024-04-26T01:08:53.573 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0979999999999998e-06 2024-04-26T01:08:53.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.573 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.573 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361923+0000", 2024-04-26T01:08:53.573 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9719999999999997e-06 2024-04-26T01:08:53.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.573 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.573 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361927+0000", 2024-04-26T01:08:53.573 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.5070000000000002e-06 2024-04-26T01:08:53.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.574 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.574 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.364059+0000", 2024-04-26T01:08:53.574 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.002132278 2024-04-26T01:08:53.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.574 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:53.574 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.364206+0000", 2024-04-26T01:08:53.574 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014703700000000001 2024-04-26T01:08:53.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.574 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.575 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473123+0000", 2024-04-26T01:08:53.575 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.108916164 2024-04-26T01:08:53.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.575 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:53.575 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473211+0000", 2024-04-26T01:08:53.575 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8381999999999995e-05 2024-04-26T01:08:53.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.575 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.575 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.476007+0000", 2024-04-26T01:08:53.575 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0027959970000000002 2024-04-26T01:08:53.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.575 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.576 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWriteReply(3.es0 33/30 ECSubWriteReply(tid=5, last_complete=33'4, committed=1, applied=1))", 2024-04-26T01:08:53.576 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.365083+0000", 2024-04-26T01:08:53.576 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 54.191197385999999, 2024-04-26T01:08:53.576 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.107988984, 2024-04-26T01:08:53.576 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.576 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.576 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:53.576 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.576 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.576 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365083+0000", 2024-04-26T01:08:53.577 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.577 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.577 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365083+0000", 2024-04-26T01:08:53.577 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.577 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.577 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365084+0000", 2024-04-26T01:08:53.577 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.9299999999999999e-07 2024-04-26T01:08:53.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.577 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.577 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365086+0000", 2024-04-26T01:08:53.578 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5239999999999999e-06 2024-04-26T01:08:53.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.578 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.578 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472672+0000", 2024-04-26T01:08:53.578 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.10758553899999999 2024-04-26T01:08:53.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.578 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.578 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472680+0000", 2024-04-26T01:08:53.578 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.6820000000000003e-06 2024-04-26T01:08:53.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.579 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.579 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472900+0000", 2024-04-26T01:08:53.579 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00022041899999999999 2024-04-26T01:08:53.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.579 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.579 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.579 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473072+0000", 2024-04-26T01:08:53.579 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017242700000000001 2024-04-26T01:08:53.579 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.579 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.579 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.579 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.579 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.2s1 33/30 ECSubWrite(tid=165, reqid=client.4521.0:80, at_version=33'6, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:08:53.580 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.391943+0000", 2024-04-26T01:08:53.580 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 54.164337707999998, 2024-04-26T01:08:53.580 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.042848432999999998, 2024-04-26T01:08:53.580 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.580 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.580 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:53.580 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.580 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.580 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.580 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391943+0000", 2024-04-26T01:08:53.580 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.580 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.580 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.581 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391943+0000", 2024-04-26T01:08:53.581 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.581 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.581 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.581 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391944+0000", 2024-04-26T01:08:53.581 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8199999999999998e-07 2024-04-26T01:08:53.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.581 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.581 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.581 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391961+0000", 2024-04-26T01:08:53.581 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6546999999999999e-05 2024-04-26T01:08:53.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.582 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.582 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.582 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.430981+0000", 2024-04-26T01:08:53.582 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.039020817999999999 2024-04-26T01:08:53.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.582 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.582 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.582 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.430991+0000", 2024-04-26T01:08:53.582 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7059999999999999e-06 2024-04-26T01:08:53.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.582 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.582 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.582 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.431059+0000", 2024-04-26T01:08:53.582 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.8198e-05 2024-04-26T01:08:53.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.583 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:53.583 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.431067+0000", 2024-04-26T01:08:53.583 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.8930000000000005e-06 2024-04-26T01:08:53.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.583 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:53.583 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.434752+0000", 2024-04-26T01:08:53.583 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0036849249999999999 2024-04-26T01:08:53.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.583 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.583 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.434792+0000", 2024-04-26T01:08:53.584 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.9464e-05 2024-04-26T01:08:53.584 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.584 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.584 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.584 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.3s4 33/30 ECSubWrite(tid=170, reqid=client.4521.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=33'2))", 2024-04-26T01:08:53.584 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.441271+0000", 2024-04-26T01:08:53.584 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 54.115009929999999, 2024-04-26T01:08:53.584 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.063989213000000003, 2024-04-26T01:08:53.584 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.584 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.584 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:53.584 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.585 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.585 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441271+0000", 2024-04-26T01:08:53.585 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.585 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.585 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441271+0000", 2024-04-26T01:08:53.585 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.585 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.585 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441272+0000", 2024-04-26T01:08:53.586 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1650000000000001e-06 2024-04-26T01:08:53.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.586 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.586 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.586 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441286+0000", 2024-04-26T01:08:53.586 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3771000000000001e-05 2024-04-26T01:08:53.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.586 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.586 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.586 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504639+0000", 2024-04-26T01:08:53.586 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.063352787999999993 2024-04-26T01:08:53.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.586 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.586 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.587 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504647+0000", 2024-04-26T01:08:53.587 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9389999999999997e-06 2024-04-26T01:08:53.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.587 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.587 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504715+0000", 2024-04-26T01:08:53.587 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7926e-05 2024-04-26T01:08:53.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.587 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:53.587 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504722+0000", 2024-04-26T01:08:53.587 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9449999999999993e-06 2024-04-26T01:08:53.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.588 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.588 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:53.588 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.505203+0000", 2024-04-26T01:08:53.588 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00048092399999999998 2024-04-26T01:08:53.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.588 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.588 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.588 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.505260+0000", 2024-04-26T01:08:53.588 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.6755000000000001e-05 2024-04-26T01:08:53.588 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.588 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.588 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.588 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.589 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:233 3.es0 3:7b1f3e32:::smithi16428812-8:head [copy-from ver 4] snapc 3=[2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-26T01:08:53.589 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:02.464122+0000", 2024-04-26T01:08:53.589 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 51.092159287999998, 2024-04-26T01:08:53.589 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.66639822699999995, 2024-04-26T01:08:53.589 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.589 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.589 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:53.589 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:53.589 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:53.589 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:53.589 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 233 2024-04-26T01:08:53.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.589 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.589 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.590 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.590 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464122+0000", 2024-04-26T01:08:53.590 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.590 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.590 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464122+0000", 2024-04-26T01:08:53.590 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.590 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.590 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464125+0000", 2024-04-26T01:08:53.590 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0939999999999999e-06 2024-04-26T01:08:53.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.591 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.591 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464136+0000", 2024-04-26T01:08:53.591 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1790000000000001e-05 2024-04-26T01:08:53.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.591 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.591 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464160+0000", 2024-04-26T01:08:53.591 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3683e-05 2024-04-26T01:08:53.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.591 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.591 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464174+0000", 2024-04-26T01:08:53.592 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.382e-05 2024-04-26T01:08:53.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.592 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.592 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464324+0000", 2024-04-26T01:08:53.592 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014991900000000001 2024-04-26T01:08:53.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.592 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:53.592 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464485+0000", 2024-04-26T01:08:53.592 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016099500000000001 2024-04-26T01:08:53.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.592 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:53.593 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480641+0000", 2024-04-26T01:08:53.593 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016155748000000001 2024-04-26T01:08:53.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.593 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:53.593 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.485807+0000", 2024-04-26T01:08:53.593 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0051663380000000004 2024-04-26T01:08:53.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.593 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:53.593 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.130287+0000", 2024-04-26T01:08:53.593 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.64448033500000002 2024-04-26T01:08:53.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.594 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.594 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.130520+0000", 2024-04-26T01:08:53.594 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00023250500000000001 2024-04-26T01:08:53.594 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.594 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.594 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.594 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.594 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:234 3.es0 3:7b1f3e32:::smithi16428812-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-04-26T01:08:53.594 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:02.464220+0000", 2024-04-26T01:08:53.594 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 51.092060945, 2024-04-26T01:08:53.594 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.019428925, 2024-04-26T01:08:53.594 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.595 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.595 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:53.595 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:53.595 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:53.595 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:53.595 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 234 2024-04-26T01:08:53.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.595 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.595 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.595 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464220+0000", 2024-04-26T01:08:53.595 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.596 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.596 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464220+0000", 2024-04-26T01:08:53.596 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.596 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.596 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464221+0000", 2024-04-26T01:08:53.596 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4500000000000001e-06 2024-04-26T01:08:53.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.596 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.596 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464229+0000", 2024-04-26T01:08:53.596 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.6229999999999997e-06 2024-04-26T01:08:53.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.597 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.597 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464250+0000", 2024-04-26T01:08:53.597 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1163e-05 2024-04-26T01:08:53.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.597 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.597 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464267+0000", 2024-04-26T01:08:53.597 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.668e-05 2024-04-26T01:08:53.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.597 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.597 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464815+0000", 2024-04-26T01:08:53.597 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00054803000000000002 2024-04-26T01:08:53.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.598 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.598 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for blocked object", 2024-04-26T01:08:53.598 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464939+0000", 2024-04-26T01:08:53.598 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000124385 2024-04-26T01:08:53.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.598 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.598 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.598 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.483264+0000", 2024-04-26T01:08:53.598 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.018324427000000001 2024-04-26T01:08:53.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.598 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.598 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:53.599 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.483429+0000", 2024-04-26T01:08:53.599 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000165749 2024-04-26T01:08:53.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.599 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.599 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.483649+0000", 2024-04-26T01:08:53.599 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00021941800000000001 2024-04-26T01:08:53.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.599 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.599 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:257 3.es0 3:7b1f3e32:::smithi16428812-8:head [append 0~0] snapc 4=[4,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:53.599 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:03.446919+0000", 2024-04-26T01:08:53.600 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 50.109361978999999, 2024-04-26T01:08:53.600 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.84360518900000003, 2024-04-26T01:08:53.600 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.600 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.600 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:53.600 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:53.600 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:53.600 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:53.600 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 257 2024-04-26T01:08:53.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.600 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.600 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.600 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446919+0000", 2024-04-26T01:08:53.601 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.601 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.601 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446919+0000", 2024-04-26T01:08:53.601 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.601 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.601 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446920+0000", 2024-04-26T01:08:53.601 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4700000000000001e-07 2024-04-26T01:08:53.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.601 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.602 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446922+0000", 2024-04-26T01:08:53.602 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6800000000000002e-06 2024-04-26T01:08:53.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.602 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.602 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446931+0000", 2024-04-26T01:08:53.602 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8659999999999995e-06 2024-04-26T01:08:53.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.602 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.602 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446941+0000", 2024-04-26T01:08:53.602 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.5540000000000003e-06 2024-04-26T01:08:53.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.458756+0000", 2024-04-26T01:08:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011815275 2024-04-26T01:08:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.459005+0000", 2024-04-26T01:08:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000249418 2024-04-26T01:08:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.460129+0000", 2024-04-26T01:08:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001123768 2024-04-26T01:08:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461504+0000", 2024-04-26T01:08:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001375079 2024-04-26T01:08:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.290324+0000", 2024-04-26T01:08:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.82881998700000004 2024-04-26T01:08:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.290524+0000", 2024-04-26T01:08:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000199815 2024-04-26T01:08:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/30 ECSubWriteReply(tid=25, last_complete=37'19, committed=1, applied=1))", 2024-04-26T01:08:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:03.461054+0000", 2024-04-26T01:08:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 50.095226990999997, 2024-04-26T01:08:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.83058926899999996, 2024-04-26T01:08:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461054+0000", 2024-04-26T01:08:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461054+0000", 2024-04-26T01:08:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461056+0000", 2024-04-26T01:08:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.906e-06 2024-04-26T01:08:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461069+0000", 2024-04-26T01:08:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.295e-05 2024-04-26T01:08:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.289919+0000", 2024-04-26T01:08:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.82884987300000001 2024-04-26T01:08:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.289933+0000", 2024-04-26T01:08:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4430000000000001e-05 2024-04-26T01:08:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.290082+0000", 2024-04-26T01:08:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014943899999999999 2024-04-26T01:08:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.291643+0000", 2024-04-26T01:08:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001560671 2024-04-26T01:08:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:385 3.fs1 3.db40cadf (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.452224+0000", 2024-04-26T01:08:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 44.104057240000003, 2024-04-26T01:08:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95265389199999995, 2024-04-26T01:08:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 385 2024-04-26T01:08:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452224+0000", 2024-04-26T01:08:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452224+0000", 2024-04-26T01:08:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452227+0000", 2024-04-26T01:08:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6830000000000001e-06 2024-04-26T01:08:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452242+0000", 2024-04-26T01:08:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4493e-05 2024-04-26T01:08:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452274+0000", 2024-04-26T01:08:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1806e-05 2024-04-26T01:08:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452300+0000", 2024-04-26T01:08:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6092999999999999e-05 2024-04-26T01:08:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452469+0000", 2024-04-26T01:08:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000169629 2024-04-26T01:08:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:08:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452475+0000", 2024-04-26T01:08:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.604e-06 2024-04-26T01:08:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.404853+0000", 2024-04-26T01:08:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95237828400000002 2024-04-26T01:08:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.404878+0000", 2024-04-26T01:08:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4300000000000001e-05 2024-04-26T01:08:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:386 3.fs1 3.db40cadf (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.452348+0000", 2024-04-26T01:08:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 44.103933038000001, 2024-04-26T01:08:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95267636899999997, 2024-04-26T01:08:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 386 2024-04-26T01:08:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452348+0000", 2024-04-26T01:08:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.616 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452348+0000", 2024-04-26T01:08:53.616 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.616 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.616 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.616 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452349+0000", 2024-04-26T01:08:53.616 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.533e-06 2024-04-26T01:08:53.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.616 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.616 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.616 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452359+0000", 2024-04-26T01:08:53.616 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.5070000000000006e-06 2024-04-26T01:08:53.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.617 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.617 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.617 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452381+0000", 2024-04-26T01:08:53.617 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2306e-05 2024-04-26T01:08:53.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.617 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.617 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.617 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452398+0000", 2024-04-26T01:08:53.617 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6456e-05 2024-04-26T01:08:53.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.617 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.617 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.617 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452605+0000", 2024-04-26T01:08:53.618 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00020750900000000001 2024-04-26T01:08:53.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.618 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.618 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:08:53.618 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452610+0000", 2024-04-26T01:08:53.618 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3569999999999997e-06 2024-04-26T01:08:53.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.618 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.618 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.618 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.404988+0000", 2024-04-26T01:08:53.618 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95237776100000004 2024-04-26T01:08:53.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.618 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.619 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.619 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405024+0000", 2024-04-26T01:08:53.619 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5939999999999998e-05 2024-04-26T01:08:53.619 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.619 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.619 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.619 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.619 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:387 3.fs1 3.db40cadf (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:53.619 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.452423+0000", 2024-04-26T01:08:53.619 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 44.103857632, 2024-04-26T01:08:53.619 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95294480000000004, 2024-04-26T01:08:53.619 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.619 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.620 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:53.620 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:53.620 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:53.620 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:53.620 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 387 2024-04-26T01:08:53.620 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.620 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.620 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.620 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.620 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452423+0000", 2024-04-26T01:08:53.620 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.620 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.620 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.620 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.621 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452423+0000", 2024-04-26T01:08:53.621 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.621 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.621 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.621 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.621 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452425+0000", 2024-04-26T01:08:53.621 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.373e-06 2024-04-26T01:08:53.621 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.621 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.621 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.621 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452435+0000", 2024-04-26T01:08:53.621 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0149999999999999e-05 2024-04-26T01:08:53.621 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.621 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.622 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.622 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452447+0000", 2024-04-26T01:08:53.622 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.199e-05 2024-04-26T01:08:53.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.622 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.622 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.622 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452461+0000", 2024-04-26T01:08:53.622 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4503e-05 2024-04-26T01:08:53.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.622 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.622 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.622 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452685+0000", 2024-04-26T01:08:53.622 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00022367200000000001 2024-04-26T01:08:53.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.623 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.623 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:08:53.623 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452687+0000", 2024-04-26T01:08:53.623 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.446e-06 2024-04-26T01:08:53.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.623 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.623 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.623 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405319+0000", 2024-04-26T01:08:53.623 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.952631279 2024-04-26T01:08:53.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.623 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.623 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.623 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405368+0000", 2024-04-26T01:08:53.623 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9387e-05 2024-04-26T01:08:53.624 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.624 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.624 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.624 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.624 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:393 3.fs1 3.94f9353f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:53.624 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.464778+0000", 2024-04-26T01:08:53.624 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 44.091503103999997, 2024-04-26T01:08:53.624 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94080343600000005, 2024-04-26T01:08:53.624 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.624 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.624 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:53.624 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:53.624 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:53.624 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:53.625 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 393 2024-04-26T01:08:53.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.625 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.625 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.625 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.625 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464778+0000", 2024-04-26T01:08:53.625 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.625 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.625 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.625 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464778+0000", 2024-04-26T01:08:53.625 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.625 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.625 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.626 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464780+0000", 2024-04-26T01:08:53.626 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2460000000000002e-06 2024-04-26T01:08:53.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.626 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.626 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.626 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464791+0000", 2024-04-26T01:08:53.626 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.145e-05 2024-04-26T01:08:53.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.626 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.626 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.626 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464823+0000", 2024-04-26T01:08:53.626 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1157e-05 2024-04-26T01:08:53.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464837+0000", 2024-04-26T01:08:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.488e-05 2024-04-26T01:08:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464985+0000", 2024-04-26T01:08:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000147999 2024-04-26T01:08:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:08:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464991+0000", 2024-04-26T01:08:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3329999999999996e-06 2024-04-26T01:08:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.628 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.628 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.628 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405522+0000", 2024-04-26T01:08:53.628 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94053098899999998 2024-04-26T01:08:53.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.628 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.628 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.628 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405581+0000", 2024-04-26T01:08:53.628 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9382000000000002e-05 2024-04-26T01:08:53.628 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.628 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.628 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.629 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.629 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:394 3.fs1 3.94f9353f (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:08:53.629 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.464870+0000", 2024-04-26T01:08:53.629 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 44.091410828000001, 2024-04-26T01:08:53.629 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94122545000000002, 2024-04-26T01:08:53.629 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.629 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.629 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:53.629 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:53.629 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:53.629 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:53.629 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 394 2024-04-26T01:08:53.629 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.629 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.630 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.630 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.630 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464870+0000", 2024-04-26T01:08:53.630 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.630 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.630 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.630 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464870+0000", 2024-04-26T01:08:53.630 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.630 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.630 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.630 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464871+0000", 2024-04-26T01:08:53.630 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1230000000000001e-06 2024-04-26T01:08:53.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.631 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.631 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.631 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464875+0000", 2024-04-26T01:08:53.631 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.25e-06 2024-04-26T01:08:53.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.631 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.631 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.631 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464895+0000", 2024-04-26T01:08:53.631 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9332999999999999e-05 2024-04-26T01:08:53.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.631 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.631 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.631 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464904+0000", 2024-04-26T01:08:53.632 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9230000000000008e-06 2024-04-26T01:08:53.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.632 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.632 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.465051+0000", 2024-04-26T01:08:53.632 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000147186 2024-04-26T01:08:53.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.632 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:08:53.632 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.465054+0000", 2024-04-26T01:08:53.632 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7530000000000002e-06 2024-04-26T01:08:53.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.633 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.633 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.406060+0000", 2024-04-26T01:08:53.633 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94100597900000005 2024-04-26T01:08:53.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.633 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.633 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.633 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.406095+0000", 2024-04-26T01:08:53.633 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5902999999999998e-05 2024-04-26T01:08:53.633 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.633 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.633 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.633 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.633 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpRead(3.fs1 58/44 ECSubRead(tid=311, to_read={3:fa21e776:::smithi16428812-18:head=0,4096,0}, subchunks={3:fa21e776:::smithi16428812-18:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:08:53.634 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:24.214139+0000", 2024-04-26T01:08:53.634 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 29.342141687000002, 2024-04-26T01:08:53.634 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.068278328999999999, 2024-04-26T01:08:53.634 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.634 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.634 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:53.634 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.634 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.634 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.634 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.214139+0000", 2024-04-26T01:08:53.634 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.634 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.635 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.635 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.214139+0000", 2024-04-26T01:08:53.635 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.635 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.635 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.635 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.214141+0000", 2024-04-26T01:08:53.635 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.674e-06 2024-04-26T01:08:53.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.635 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.635 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.635 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.214161+0000", 2024-04-26T01:08:53.635 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0349000000000001e-05 2024-04-26T01:08:53.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.636 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.636 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.636 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.281949+0000", 2024-04-26T01:08:53.636 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.067788082999999999 2024-04-26T01:08:53.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.636 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.636 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.636 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.281978+0000", 2024-04-26T01:08:53.636 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8646999999999999e-05 2024-04-26T01:08:53.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.636 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.636 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.636 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.282180+0000", 2024-04-26T01:08:53.636 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00020251500000000001 2024-04-26T01:08:53.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.637 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.637 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.637 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.282417+0000", 2024-04-26T01:08:53.637 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000237061 2024-04-26T01:08:53.637 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.637 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.637 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.637 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.637 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:692 3.es0 3:7eef64f7:::smithi16428812-21:head [read 0~2719744,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-26T01:08:53.637 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:30.194619+0000", 2024-04-26T01:08:53.637 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 23.361661392999999, 2024-04-26T01:08:53.637 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.021611364000000001, 2024-04-26T01:08:53.638 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.638 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.638 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:53.638 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:53.638 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:53.638 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:53.638 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 692 2024-04-26T01:08:53.638 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.638 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.638 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.638 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.638 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194619+0000", 2024-04-26T01:08:53.638 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194619+0000", 2024-04-26T01:08:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194620+0000", 2024-04-26T01:08:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7300000000000004e-07 2024-04-26T01:08:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194641+0000", 2024-04-26T01:08:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0910000000000001e-05 2024-04-26T01:08:53.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.640 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.640 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.640 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194653+0000", 2024-04-26T01:08:53.640 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1663e-05 2024-04-26T01:08:53.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.640 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.640 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.640 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194661+0000", 2024-04-26T01:08:53.640 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.49e-06 2024-04-26T01:08:53.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.640 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.640 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.640 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194940+0000", 2024-04-26T01:08:53.640 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00027803799999999998 2024-04-26T01:08:53.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.641 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.641 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:53.641 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194956+0000", 2024-04-26T01:08:53.641 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6702999999999999e-05 2024-04-26T01:08:53.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.641 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.641 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.641 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.201697+0000", 2024-04-26T01:08:53.641 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0067405629999999998 2024-04-26T01:08:53.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.641 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.641 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:53.641 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.201779+0000", 2024-04-26T01:08:53.641 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2678999999999999e-05 2024-04-26T01:08:53.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.642 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.642 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.642 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.216231+0000", 2024-04-26T01:08:53.642 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014451345000000001 2024-04-26T01:08:53.642 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.642 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.642 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.642 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.642 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:902 3.es0 3:7b1f3e32:::smithi16428812-8:head [copy-from ver 32] snapc 1b=[1b,1a,19,18] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:08:53.642 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:47.482197+0000", 2024-04-26T01:08:53.642 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 6.074083602, 2024-04-26T01:08:53.642 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.89556173800000005, 2024-04-26T01:08:53.643 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.643 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.643 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:53.643 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:53.643 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:53.643 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:53.643 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 902 2024-04-26T01:08:53.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.643 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.643 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.643 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.643 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482197+0000", 2024-04-26T01:08:53.643 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482197+0000", 2024-04-26T01:08:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482199+0000", 2024-04-26T01:08:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5710000000000001e-06 2024-04-26T01:08:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482205+0000", 2024-04-26T01:08:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.489e-06 2024-04-26T01:08:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.645 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.645 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.645 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482218+0000", 2024-04-26T01:08:53.645 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2996000000000001e-05 2024-04-26T01:08:53.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.645 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.645 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.645 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482226+0000", 2024-04-26T01:08:53.645 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5510000000000002e-06 2024-04-26T01:08:53.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.645 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.645 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.645 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482371+0000", 2024-04-26T01:08:53.646 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014525599999999999 2024-04-26T01:08:53.646 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.646 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.646 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:53.646 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482652+0000", 2024-04-26T01:08:53.646 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000281165 2024-04-26T01:08:53.646 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.646 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.646 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:53.646 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.367066+0000", 2024-04-26T01:08:53.646 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.88441396500000002 2024-04-26T01:08:53.646 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.646 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.646 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:53.647 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.375685+0000", 2024-04-26T01:08:53.647 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0086183449999999995 2024-04-26T01:08:53.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.647 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.647 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:53.647 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.377540+0000", 2024-04-26T01:08:53.647 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001855659 2024-04-26T01:08:53.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.647 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.647 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.647 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.377759+0000", 2024-04-26T01:08:53.647 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000218741 2024-04-26T01:08:53.647 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.647 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.647 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.648 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.648 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:903 3.es0 3:7b1f3e32:::smithi16428812-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:08:53.648 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:47.482246+0000", 2024-04-26T01:08:53.648 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 6.074034385, 2024-04-26T01:08:53.648 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.89304915500000004, 2024-04-26T01:08:53.648 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:53.648 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:53.648 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:53.648 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:53.648 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:53.648 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:53.648 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 903 2024-04-26T01:08:53.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.649 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:53.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.649 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:53.649 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482246+0000", 2024-04-26T01:08:53.649 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.649 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:53.649 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482246+0000", 2024-04-26T01:08:53.649 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:53.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.649 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:53.649 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482247+0000", 2024-04-26T01:08:53.649 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7500000000000001e-07 2024-04-26T01:08:53.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.650 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:53.650 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482249+0000", 2024-04-26T01:08:53.650 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8840000000000001e-06 2024-04-26T01:08:53.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.650 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:53.650 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482253+0000", 2024-04-26T01:08:53.650 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.4440000000000001e-06 2024-04-26T01:08:53.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.650 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:53.650 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482256+0000", 2024-04-26T01:08:53.651 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1700000000000001e-06 2024-04-26T01:08:53.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.651 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.651 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.483084+0000", 2024-04-26T01:08:53.651 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00082729500000000003 2024-04-26T01:08:53.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.651 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for blocked object", 2024-04-26T01:08:53.651 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.483147+0000", 2024-04-26T01:08:53.651 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.3812999999999994e-05 2024-04-26T01:08:53.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.651 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:53.651 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.374930+0000", 2024-04-26T01:08:53.652 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.89178298199999995 2024-04-26T01:08:53.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.652 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.652 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:53.652 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.375134+0000", 2024-04-26T01:08:53.652 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00020325599999999999 2024-04-26T01:08:53.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:53.652 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:53.652 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:53.652 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.375296+0000", 2024-04-26T01:08:53.652 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000161936 2024-04-26T01:08:53.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.652 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.653 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:53.653 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:53.653 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:53.654 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:53.654 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:53.655 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:53.596+0000 7fa308b5b640 0 -- 172.21.15.8:0/1527212249 >> [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] conn(0x7fa304223230 msgr2=0x7fa304225620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-26T01:08:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T01:08:53.656 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:08:53.656 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:53.651+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:53.656 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:53.657 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T01:08:53.755 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:53.753+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:53.809 INFO:teuthology.orchestra.run.smithi164.stdout:get_pg_num_history pg_num_history pg_num_history(e79 pg_nums {1={23=1},2={25=8},3={30=16}} deleted_pools ) 2024-04-26T01:08:53.809 INFO:teuthology.orchestra.run.smithi164.stdout:Importing pgid 3.3s6 2024-04-26T01:08:53.809 INFO:teuthology.orchestra.run.smithi164.stdout:Write 6#3:c992c93e:::smithi16428812-37:15# 2024-04-26T01:08:53.812 INFO:teuthology.orchestra.run.smithi164.stdout:Write 6#3:c992c93e:::smithi16428812-37:head# 2024-04-26T01:08:53.812 INFO:teuthology.orchestra.run.smithi164.stdout:snapset 15=[]:{15=[15]} 2024-04-26T01:08:53.814 INFO:teuthology.orchestra.run.smithi164.stdout:write_pg epoch 76 info 3.3s6( v 72'20 (0'0,72'20] local-lis/les=30/31 n=2 ec=30/30 lis/c=30/30 les/c/f=31/31/0 sis=30) 2024-04-26T01:08:53.814 INFO:teuthology.orchestra.run.smithi164.stdout:Import successful 2024-04-26T01:08:53.826 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:53.826 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:53.826 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:08:53.826 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:53.840 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:08:53.856 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:53.853+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:53.958 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:08:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:08:53.958 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:53.958 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:53.956+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:53.971 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:08:54.051 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-04-26T01:08:54.056 INFO:tasks.rados.rados.0.smithi164.stdout:377: done (1 left) 2024-04-26T01:08:54.056 INFO:tasks.rados.rados.0.smithi164.stdout:378: rollback oid 41 current snap is 29 2024-04-26T01:08:54.056 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 41 to 27 2024-04-26T01:08:54.056 INFO:tasks.rados.rados.0.smithi164.stdout:376: finishing write tid 1 to smithi16428812-32 2024-04-26T01:08:54.056 INFO:tasks.rados.rados.0.smithi164.stdout:376: finishing write tid 2 to smithi16428812-32 2024-04-26T01:08:54.056 INFO:tasks.rados.rados.0.smithi164.stdout:376: finishing write tid 3 to smithi16428812-32 2024-04-26T01:08:54.057 INFO:tasks.rados.rados.0.smithi164.stdout:376: oid 32 updating version 0 to 69 2024-04-26T01:08:54.057 INFO:tasks.rados.rados.0.smithi164.stdout:376: oid 32 version 69 is already newer than 68 2024-04-26T01:08:54.057 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 32 v 69 (ObjNum 152 snap 29 seq_num 152) dirty exists 2024-04-26T01:08:54.057 INFO:tasks.rados.rados.0.smithi164.stdout:376: left oid 32 (ObjNum 152 snap 29 seq_num 152) 2024-04-26T01:08:54.057 INFO:tasks.rados.rados.0.smithi164.stdout:376: done (1 left) 2024-04-26T01:08:54.057 INFO:tasks.rados.rados.0.smithi164.stdout:379: copy_from oid 23 from oid 13 current snap is 29 2024-04-26T01:08:54.057 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 22 current snap is 29 2024-04-26T01:08:54.057 INFO:tasks.rados.rados.0.smithi164.stdout:380: seq_num 153 ranges {0=163840} 2024-04-26T01:08:54.058 INFO:tasks.rados.rados.0.smithi164.stdout:380: writing smithi16428812-22 from 0 to 163840 tid 1 2024-04-26T01:08:54.058 INFO:tasks.rados.rados.0.smithi164.stdout:381: rmattr oid 21 current snap is 29 2024-04-26T01:08:54.058 INFO:tasks.rados.rados.0.smithi164.stdout:381: done (3 left) 2024-04-26T01:08:54.059 INFO:tasks.rados.rados.0.smithi164.stdout:382: snap_create 2024-04-26T01:08:54.059 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:54.053+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:54.073 INFO:tasks.rados.rados.0.smithi164.stdout:378: finishing rollback tid 0 to smithi16428812-41 2024-04-26T01:08:54.075 INFO:tasks.rados.rados.0.smithi164.stdout:380: finishing write tid 1 to smithi16428812-22 2024-04-26T01:08:54.076 INFO:tasks.rados.rados.0.smithi164.stdout:380: finishing write tid 2 to smithi16428812-22 2024-04-26T01:08:54.077 INFO:tasks.rados.rados.0.smithi164.stdout:378: finishing rollback tid 1 to smithi16428812-41 2024-04-26T01:08:54.077 INFO:tasks.rados.rados.0.smithi164.stdout:378: finishing rollback tid 2 to smithi16428812-41 2024-04-26T01:08:54.077 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 41 v 84 (ObjNum 137 snap 21 seq_num 137) dirty exists 2024-04-26T01:08:54.081 INFO:tasks.rados.rados.0.smithi164.stdout:379: finishing copy_from racing read to smithi16428812-23 2024-04-26T01:08:54.082 INFO:tasks.rados.rados.0.smithi164.stdout:380: finishing write tid 3 to smithi16428812-22 2024-04-26T01:08:54.082 INFO:tasks.rados.rados.0.smithi164.stdout:380: oid 22 updating version 0 to 98 2024-04-26T01:08:54.082 INFO:tasks.rados.rados.0.smithi164.stdout:380: oid 22 updating version 98 to 99 2024-04-26T01:08:54.082 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 22 v 99 (ObjNum 153 snap 29 seq_num 153) dirty exists 2024-04-26T01:08:54.082 INFO:tasks.rados.rados.0.smithi164.stdout:380: left oid 22 (ObjNum 153 snap 29 seq_num 153) 2024-04-26T01:08:54.085 INFO:tasks.rados.rados.0.smithi164.stdout:379: finishing copy_from to smithi16428812-23 2024-04-26T01:08:54.085 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 23 v 97 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-26T01:08:54.108 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:08:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:08:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:08:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 36/30 ECSubWrite(tid=15, reqid=client.4521.0:233, at_version=36'11, trim_to=0'0, roll_forward_to=33'9))", 2024-04-26T01:08:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:02.480147+0000", 2024-04-26T01:08:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 51.627145796000001, 2024-04-26T01:08:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.64949222799999995, 2024-04-26T01:08:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480147+0000", 2024-04-26T01:08:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480147+0000", 2024-04-26T01:08:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480148+0000", 2024-04-26T01:08:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2300000000000001e-06 2024-04-26T01:08:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480517+0000", 2024-04-26T01:08:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00036900700000000003 2024-04-26T01:08:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125262+0000", 2024-04-26T01:08:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.64474460600000005 2024-04-26T01:08:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125278+0000", 2024-04-26T01:08:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5622999999999999e-05 2024-04-26T01:08:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125483+0000", 2024-04-26T01:08:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000205105 2024-04-26T01:08:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125502+0000", 2024-04-26T01:08:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8876999999999999e-05 2024-04-26T01:08:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.129565+0000", 2024-04-26T01:08:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0040633930000000002 2024-04-26T01:08:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.129639+0000", 2024-04-26T01:08:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4387000000000005e-05 2024-04-26T01:08:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.5s6 43/30 ECSubWrite(tid=35, reqid=osd.15.0:35, at_version=43'30, trim_to=0'0, roll_forward_to=42'28))", 2024-04-26T01:08:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 44.700734056000002, 2024-04-26T01:08:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011310863000000001, 2024-04-26T01:08:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:08:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2e-07 2024-04-26T01:08:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406612+0000", 2024-04-26T01:08:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2206000000000001e-05 2024-04-26T01:08:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406709+0000", 2024-04-26T01:08:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7106000000000002e-05 2024-04-26T01:08:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406737+0000", 2024-04-26T01:08:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8629999999999999e-05 2024-04-26T01:08:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.407351+0000", 2024-04-26T01:08:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00061410800000000004 2024-04-26T01:08:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.407377+0000", 2024-04-26T01:08:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5057000000000001e-05 2024-04-26T01:08:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.417771+0000", 2024-04-26T01:08:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.01039402 2024-04-26T01:08:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.417870+0000", 2024-04-26T01:08:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9215999999999993e-05 2024-04-26T01:08:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 50/44 ECSubWrite(tid=51, reqid=osd.2.0:51, at_version=50'40, trim_to=0'0, roll_forward_to=45'38))", 2024-04-26T01:08:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.058137+0000", 2024-04-26T01:08:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.049155673999998, 2024-04-26T01:08:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0092019839999999999, 2024-04-26T01:08:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058137+0000", 2024-04-26T01:08:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058137+0000", 2024-04-26T01:08:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058138+0000", 2024-04-26T01:08:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.3300000000000001e-07 2024-04-26T01:08:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058197+0000", 2024-04-26T01:08:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9110000000000002e-05 2024-04-26T01:08:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058307+0000", 2024-04-26T01:08:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000109706 2024-04-26T01:08:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058330+0000", 2024-04-26T01:08:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3540000000000002e-05 2024-04-26T01:08:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058521+0000", 2024-04-26T01:08:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019083500000000001 2024-04-26T01:08:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058548+0000", 2024-04-26T01:08:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6950000000000001e-05 2024-04-26T01:08:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.067237+0000", 2024-04-26T01:08:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0086888769999999994 2024-04-26T01:08:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.067339+0000", 2024-04-26T01:08:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010213300000000001 2024-04-26T01:08:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs4 50/44 ECSubWrite(tid=22, reqid=osd.9.0:22, at_version=50'37, trim_to=0'0, roll_forward_to=47'35))", 2024-04-26T01:08:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:08:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.048084355, 2024-04-26T01:08:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016916468, 2024-04-26T01:08:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:08:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:08:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:08:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9299999999999996e-07 2024-04-26T01:08:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059282+0000", 2024-04-26T01:08:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2019000000000003e-05 2024-04-26T01:08:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059397+0000", 2024-04-26T01:08:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000115569 2024-04-26T01:08:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059419+0000", 2024-04-26T01:08:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1497000000000001e-05 2024-04-26T01:08:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064015+0000", 2024-04-26T01:08:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0045966059999999996 2024-04-26T01:08:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064041+0000", 2024-04-26T01:08:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5570000000000001e-05 2024-04-26T01:08:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.075972+0000", 2024-04-26T01:08:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011930802000000001 2024-04-26T01:08:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076125+0000", 2024-04-26T01:08:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015351199999999999 2024-04-26T01:08:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 50/44 ECSubWrite(tid=215, reqid=client.4521.0:490, at_version=50'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-26T01:08:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.059258+0000", 2024-04-26T01:08:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.048034483000002, 2024-04-26T01:08:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014073085000000001, 2024-04-26T01:08:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059258+0000", 2024-04-26T01:08:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059258+0000", 2024-04-26T01:08:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059261+0000", 2024-04-26T01:08:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3829999999999999e-06 2024-04-26T01:08:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059926+0000", 2024-04-26T01:08:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00066483199999999997 2024-04-26T01:08:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060011+0000", 2024-04-26T01:08:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5248999999999996e-05 2024-04-26T01:08:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060033+0000", 2024-04-26T01:08:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1566999999999999e-05 2024-04-26T01:08:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060387+0000", 2024-04-26T01:08:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000354793 2024-04-26T01:08:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060449+0000", 2024-04-26T01:08:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.1957000000000003e-05 2024-04-26T01:08:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073174+0000", 2024-04-26T01:08:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012724998 2024-04-26T01:08:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073332+0000", 2024-04-26T01:08:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015730599999999999 2024-04-26T01:08:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.0s5 50/30 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=50'32, trim_to=0'0, roll_forward_to=45'30))", 2024-04-26T01:08:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.062205+0000", 2024-04-26T01:08:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.045088102999998, 2024-04-26T01:08:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.01120789, 2024-04-26T01:08:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062205+0000", 2024-04-26T01:08:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062205+0000", 2024-04-26T01:08:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062207+0000", 2024-04-26T01:08:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.844e-06 2024-04-26T01:08:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062258+0000", 2024-04-26T01:08:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.1388999999999999e-05 2024-04-26T01:08:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062376+0000", 2024-04-26T01:08:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00011767199999999999 2024-04-26T01:08:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062399+0000", 2024-04-26T01:08:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3410000000000001e-05 2024-04-26T01:08:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062583+0000", 2024-04-26T01:08:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00018336900000000001 2024-04-26T01:08:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062611+0000", 2024-04-26T01:08:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8189999999999999e-05 2024-04-26T01:08:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073316+0000", 2024-04-26T01:08:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010704824 2024-04-26T01:08:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073413+0000", 2024-04-26T01:08:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7192000000000001e-05 2024-04-26T01:08:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.2s3 50/44 ECSubWrite(tid=212, reqid=osd.13.0:212, at_version=50'24, trim_to=0'0, roll_forward_to=45'22))", 2024-04-26T01:08:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.063661+0000", 2024-04-26T01:08:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.043631484999999, 2024-04-26T01:08:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012570563999999999, 2024-04-26T01:08:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063661+0000", 2024-04-26T01:08:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063661+0000", 2024-04-26T01:08:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063662+0000", 2024-04-26T01:08:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.37e-07 2024-04-26T01:08:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063721+0000", 2024-04-26T01:08:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.8696999999999997e-05 2024-04-26T01:08:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063860+0000", 2024-04-26T01:08:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013883799999999999 2024-04-26T01:08:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063882+0000", 2024-04-26T01:08:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2313000000000001e-05 2024-04-26T01:08:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064661+0000", 2024-04-26T01:08:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00077955699999999995 2024-04-26T01:08:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064686+0000", 2024-04-26T01:08:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4984e-05 2024-04-26T01:08:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.075967+0000", 2024-04-26T01:08:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011280646 2024-04-26T01:08:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076232+0000", 2024-04-26T01:08:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00026499200000000003 2024-04-26T01:08:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 50/30 ECSubWrite(tid=216, reqid=osd.4.0:214, at_version=0'0, trim_to=0'0, roll_forward_to=50'22))", 2024-04-26T01:08:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.065107+0000", 2024-04-26T01:08:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.042185582999998, 2024-04-26T01:08:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0087328609999999997, 2024-04-26T01:08:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065107+0000", 2024-04-26T01:08:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065107+0000", 2024-04-26T01:08:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065108+0000", 2024-04-26T01:08:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.0999999999999997e-07 2024-04-26T01:08:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065125+0000", 2024-04-26T01:08:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7010000000000001e-05 2024-04-26T01:08:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065360+0000", 2024-04-26T01:08:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00023507199999999999 2024-04-26T01:08:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065382+0000", 2024-04-26T01:08:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1993000000000001e-05 2024-04-26T01:08:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065672+0000", 2024-04-26T01:08:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00028928100000000001 2024-04-26T01:08:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065696+0000", 2024-04-26T01:08:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4273e-05 2024-04-26T01:08:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073748+0000", 2024-04-26T01:08:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0080520490000000004 2024-04-26T01:08:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073840+0000", 2024-04-26T01:08:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2373000000000005e-05 2024-04-26T01:08:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 50/44 ECSubWrite(tid=218, reqid=osd.4.0:218, at_version=50'34, trim_to=0'0, roll_forward_to=47'30))", 2024-04-26T01:08:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.065939+0000", 2024-04-26T01:08:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.041353968999999, 2024-04-26T01:08:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010065406000000001, 2024-04-26T01:08:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065939+0000", 2024-04-26T01:08:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065939+0000", 2024-04-26T01:08:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065941+0000", 2024-04-26T01:08:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9199999999999998e-06 2024-04-26T01:08:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065994+0000", 2024-04-26T01:08:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3439000000000001e-05 2024-04-26T01:08:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066088+0000", 2024-04-26T01:08:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.365e-05 2024-04-26T01:08:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066110+0000", 2024-04-26T01:08:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1912999999999998e-05 2024-04-26T01:08:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066779+0000", 2024-04-26T01:08:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00066918399999999997 2024-04-26T01:08:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066810+0000", 2024-04-26T01:08:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0446999999999998e-05 2024-04-26T01:08:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.075902+0000", 2024-04-26T01:08:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0090923529999999992 2024-04-26T01:08:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076004+0000", 2024-04-26T01:08:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0001025 2024-04-26T01:08:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.8s6 50/30 ECSubWrite(tid=42, reqid=osd.5.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=50'26))", 2024-04-26T01:08:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.068230+0000", 2024-04-26T01:08:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.039063118000001, 2024-04-26T01:08:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0080202260000000001, 2024-04-26T01:08:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068230+0000", 2024-04-26T01:08:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068230+0000", 2024-04-26T01:08:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068232+0000", 2024-04-26T01:08:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.804e-06 2024-04-26T01:08:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068261+0000", 2024-04-26T01:08:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9595999999999999e-05 2024-04-26T01:08:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068341+0000", 2024-04-26T01:08:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9302999999999996e-05 2024-04-26T01:08:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068355+0000", 2024-04-26T01:08:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.465e-05 2024-04-26T01:08:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068495+0000", 2024-04-26T01:08:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013948500000000001 2024-04-26T01:08:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068520+0000", 2024-04-26T01:08:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5015999999999999e-05 2024-04-26T01:08:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076177+0000", 2024-04-26T01:08:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0076572489999999997 2024-04-26T01:08:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076250+0000", 2024-04-26T01:08:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.3122999999999998e-05 2024-04-26T01:08:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 55/44 ECSubWrite(tid=253, reqid=osd.4.0:253, at_version=55'43, trim_to=0'0, roll_forward_to=52'41))", 2024-04-26T01:08:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:21.084283+0000", 2024-04-26T01:08:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 33.023010286999998, 2024-04-26T01:08:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061481820000000003, 2024-04-26T01:08:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084283+0000", 2024-04-26T01:08:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.155 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.155 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.155 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084283+0000", 2024-04-26T01:08:54.155 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.155 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.155 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.155 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.155 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084283+0000", 2024-04-26T01:08:54.155 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0699999999999997e-07 2024-04-26T01:08:54.155 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.155 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.155 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.155 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084362+0000", 2024-04-26T01:08:54.155 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9083000000000002e-05 2024-04-26T01:08:54.156 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.156 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.156 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.156 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084566+0000", 2024-04-26T01:08:54.156 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00020390800000000001 2024-04-26T01:08:54.156 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.156 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.156 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.156 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084589+0000", 2024-04-26T01:08:54.156 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3213e-05 2024-04-26T01:08:54.156 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.156 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.156 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.157 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084751+0000", 2024-04-26T01:08:54.157 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016103600000000001 2024-04-26T01:08:54.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.157 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.157 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.157 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084781+0000", 2024-04-26T01:08:54.157 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0372999999999999e-05 2024-04-26T01:08:54.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.157 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.157 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.157 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.090337+0000", 2024-04-26T01:08:54.157 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.005556503 2024-04-26T01:08:54.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.157 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.158 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.158 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.090431+0000", 2024-04-26T01:08:54.158 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.3459000000000001e-05 2024-04-26T01:08:54.158 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.158 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.158 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.158 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.158 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 55/30 ECSubWrite(tid=252, reqid=client.4521.0:548, at_version=55'28, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:54.158 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:21.084368+0000", 2024-04-26T01:08:54.158 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 33.022924719999999, 2024-04-26T01:08:54.158 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0068931720000000004, 2024-04-26T01:08:54.158 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.158 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.159 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.159 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.159 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.159 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.159 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084368+0000", 2024-04-26T01:08:54.159 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.159 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.159 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.159 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084368+0000", 2024-04-26T01:08:54.159 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.159 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.159 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.160 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084369+0000", 2024-04-26T01:08:54.160 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.1999999999999999e-07 2024-04-26T01:08:54.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.160 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.160 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.160 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085034+0000", 2024-04-26T01:08:54.160 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00066459099999999997 2024-04-26T01:08:54.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.160 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.160 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.160 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085069+0000", 2024-04-26T01:08:54.160 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4965999999999997e-05 2024-04-26T01:08:54.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.160 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.160 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.161 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085081+0000", 2024-04-26T01:08:54.161 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2422999999999999e-05 2024-04-26T01:08:54.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.161 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.161 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.161 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085243+0000", 2024-04-26T01:08:54.161 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000162466 2024-04-26T01:08:54.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.161 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.161 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.161 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085270+0000", 2024-04-26T01:08:54.161 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6775999999999999e-05 2024-04-26T01:08:54.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.162 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.162 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.162 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091175+0000", 2024-04-26T01:08:54.162 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059045340000000003 2024-04-26T01:08:54.162 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.162 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.162 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.162 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091261+0000", 2024-04-26T01:08:54.162 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.6695999999999998e-05 2024-04-26T01:08:54.162 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.162 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.162 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.162 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.162 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 55/44 ECSubWrite(tid=55, reqid=osd.2.0:55, at_version=55'44, trim_to=0'0, roll_forward_to=52'42))", 2024-04-26T01:08:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:21.093675+0000", 2024-04-26T01:08:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 33.013617803999999, 2024-04-26T01:08:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066519099999999996, 2024-04-26T01:08:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093675+0000", 2024-04-26T01:08:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093675+0000", 2024-04-26T01:08:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093676+0000", 2024-04-26T01:08:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.3e-07 2024-04-26T01:08:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093734+0000", 2024-04-26T01:08:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7833e-05 2024-04-26T01:08:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093836+0000", 2024-04-26T01:08:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010234900000000001 2024-04-26T01:08:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093859+0000", 2024-04-26T01:08:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3383e-05 2024-04-26T01:08:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.094252+0000", 2024-04-26T01:08:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000392424 2024-04-26T01:08:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.094287+0000", 2024-04-26T01:08:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5713000000000002e-05 2024-04-26T01:08:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.100231+0000", 2024-04-26T01:08:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059430519999999999 2024-04-26T01:08:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.100327+0000", 2024-04-26T01:08:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6526000000000004e-05 2024-04-26T01:08:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 59/30 ECSubWrite(tid=322, reqid=osd.4.0:322, at_version=59'45, trim_to=0'0, roll_forward_to=58'41))", 2024-04-26T01:08:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.160873+0000", 2024-04-26T01:08:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 28.946420278000002, 2024-04-26T01:08:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0064776979999999996, 2024-04-26T01:08:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160873+0000", 2024-04-26T01:08:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160873+0000", 2024-04-26T01:08:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160874+0000", 2024-04-26T01:08:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5030000000000001e-06 2024-04-26T01:08:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160924+0000", 2024-04-26T01:08:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0158999999999999e-05 2024-04-26T01:08:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161010+0000", 2024-04-26T01:08:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5285999999999996e-05 2024-04-26T01:08:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161032+0000", 2024-04-26T01:08:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2427000000000001e-05 2024-04-26T01:08:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161983+0000", 2024-04-26T01:08:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00095064900000000003 2024-04-26T01:08:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.162010+0000", 2024-04-26T01:08:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7786000000000001e-05 2024-04-26T01:08:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167263+0000", 2024-04-26T01:08:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0052521979999999996 2024-04-26T01:08:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167350+0000", 2024-04-26T01:08:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7689999999999996e-05 2024-04-26T01:08:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs4 70/44 ECSubWrite(tid=70, reqid=osd.9.0:70, at_version=70'78, trim_to=0'0, roll_forward_to=69'76))", 2024-04-26T01:08:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:36.069042+0000", 2024-04-26T01:08:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 18.038250807000001, 2024-04-26T01:08:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0063037789999999998, 2024-04-26T01:08:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069042+0000", 2024-04-26T01:08:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069042+0000", 2024-04-26T01:08:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069043+0000", 2024-04-26T01:08:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0699999999999997e-07 2024-04-26T01:08:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069117+0000", 2024-04-26T01:08:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4338999999999994e-05 2024-04-26T01:08:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069244+0000", 2024-04-26T01:08:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000127222 2024-04-26T01:08:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069270+0000", 2024-04-26T01:08:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5633e-05 2024-04-26T01:08:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069475+0000", 2024-04-26T01:08:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00020550200000000001 2024-04-26T01:08:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.069507+0000", 2024-04-26T01:08:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1883e-05 2024-04-26T01:08:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.075253+0000", 2024-04-26T01:08:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0057461539999999998 2024-04-26T01:08:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:36.075346+0000", 2024-04-26T01:08:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2539000000000003e-05 2024-04-26T01:08:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.5s6 72/30 ECSubWrite(tid=56, reqid=client.4521.0:785, at_version=72'49, trim_to=0'0, roll_forward_to=72'47))", 2024-04-26T01:08:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:38.301857+0000", 2024-04-26T01:08:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.805436209, 2024-04-26T01:08:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0073268409999999997, 2024-04-26T01:08:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.301857+0000", 2024-04-26T01:08:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.301857+0000", 2024-04-26T01:08:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.301859+0000", 2024-04-26T01:08:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.588e-06 2024-04-26T01:08:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.306215+0000", 2024-04-26T01:08:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0043557179999999997 2024-04-26T01:08:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.306269+0000", 2024-04-26T01:08:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.4289999999999997e-05 2024-04-26T01:08:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.306282+0000", 2024-04-26T01:08:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2332e-05 2024-04-26T01:08:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.306357+0000", 2024-04-26T01:08:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5117999999999995e-05 2024-04-26T01:08:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.306368+0000", 2024-04-26T01:08:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1063000000000001e-05 2024-04-26T01:08:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.309146+0000", 2024-04-26T01:08:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0027777040000000002 2024-04-26T01:08:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.309184+0000", 2024-04-26T01:08:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8028000000000003e-05 2024-04-26T01:08:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs4 82/44 ECSubWrite(tid=95, reqid=client.4521.0:914, at_version=82'96, trim_to=0'0, roll_forward_to=81'94))", 2024-04-26T01:08:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:48.332852+0000", 2024-04-26T01:08:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 5.7744405539999999, 2024-04-26T01:08:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066861070000000002, 2024-04-26T01:08:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.332852+0000", 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.332852+0000", 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.332853+0000", 2024-04-26T01:08:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7000000000000005e-07 2024-04-26T01:08:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.332907+0000", 2024-04-26T01:08:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3706999999999999e-05 2024-04-26T01:08:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.333021+0000", 2024-04-26T01:08:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00011459900000000001 2024-04-26T01:08:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.333045+0000", 2024-04-26T01:08:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3422999999999999e-05 2024-04-26T01:08:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.333196+0000", 2024-04-26T01:08:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000151352 2024-04-26T01:08:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.333237+0000", 2024-04-26T01:08:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.1386e-05 2024-04-26T01:08:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.339469+0000", 2024-04-26T01:08:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062311399999999996 2024-04-26T01:08:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.339539+0000", 2024-04-26T01:08:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9930000000000003e-05 2024-04-26T01:08:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 82/30 ECSubWrite(tid=490, reqid=client.4521.0:912, at_version=82'73, trim_to=0'0, roll_forward_to=81'71))", 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:48.355097+0000", 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 5.7521960979999998, 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0064842040000000004, 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.355097+0000", 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.355097+0000", 2024-04-26T01:08:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.355097+0000", 2024-04-26T01:08:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7400000000000003e-07 2024-04-26T01:08:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.356630+0000", 2024-04-26T01:08:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0015323100000000001 2024-04-26T01:08:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.356696+0000", 2024-04-26T01:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.6347999999999996e-05 2024-04-26T01:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.356714+0000", 2024-04-26T01:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7762e-05 2024-04-26T01:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.357600+0000", 2024-04-26T01:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00088573999999999999 2024-04-26T01:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.357625+0000", 2024-04-26T01:08:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5389999999999999e-05 2024-04-26T01:08:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.361520+0000", 2024-04-26T01:08:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0038951519999999998 2024-04-26T01:08:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.361581+0000", 2024-04-26T01:08:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0927999999999997e-05 2024-04-26T01:08:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 82/44 ECSubWrite(tid=81, reqid=client.4521.0:902, at_version=82'60, trim_to=0'0, roll_forward_to=81'58))", 2024-04-26T01:08:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:48.366852+0000", 2024-04-26T01:08:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 5.7404414350000001, 2024-04-26T01:08:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0074237579999999999, 2024-04-26T01:08:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.366852+0000", 2024-04-26T01:08:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.366852+0000", 2024-04-26T01:08:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.366852+0000", 2024-04-26T01:08:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5700000000000001e-07 2024-04-26T01:08:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.367807+0000", 2024-04-26T01:08:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00095427999999999995 2024-04-26T01:08:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.367877+0000", 2024-04-26T01:08:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.0500000000000006e-05 2024-04-26T01:08:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.367905+0000", 2024-04-26T01:08:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8119999999999998e-05 2024-04-26T01:08:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.368072+0000", 2024-04-26T01:08:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000166658 2024-04-26T01:08:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.368093+0000", 2024-04-26T01:08:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1290000000000001e-05 2024-04-26T01:08:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.374225+0000", 2024-04-26T01:08:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061319219999999997 2024-04-26T01:08:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.374275+0000", 2024-04-26T01:08:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0130999999999998e-05 2024-04-26T01:08:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 83/30 ECSubWrite(tid=497, reqid=osd.4.0:497, at_version=83'77, trim_to=0'0, roll_forward_to=82'73))", 2024-04-26T01:08:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:49.332826+0000", 2024-04-26T01:08:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 4.7744665489999996, 2024-04-26T01:08:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.007220565, 2024-04-26T01:08:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.332826+0000", 2024-04-26T01:08:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.332826+0000", 2024-04-26T01:08:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.332829+0000", 2024-04-26T01:08:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3130000000000001e-06 2024-04-26T01:08:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.332883+0000", 2024-04-26T01:08:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3820000000000003e-05 2024-04-26T01:08:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.332991+0000", 2024-04-26T01:08:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107972 2024-04-26T01:08:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.333015+0000", 2024-04-26T01:08:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4287e-05 2024-04-26T01:08:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.333152+0000", 2024-04-26T01:08:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013734899999999999 2024-04-26T01:08:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.333179+0000", 2024-04-26T01:08:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7202000000000001e-05 2024-04-26T01:08:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.339853+0000", 2024-04-26T01:08:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066739340000000003 2024-04-26T01:08:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.340047+0000", 2024-04-26T01:08:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019368799999999999 2024-04-26T01:08:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.197 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:54.199 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:08:54.199 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:54.197+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:54.300 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:54.296+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:54.323 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:08:54.323 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:54.337 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:08:54.349 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f /var/log/ceph/exp.3.3s6.3 2024-04-26T01:08:54.401 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:54.399+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:54.404 INFO:tasks.thrashosds.thrasher:checking if target objectstore is bluestore on osd.3 2024-04-26T01:08:54.404 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-3 show-label 2024-04-26T01:08:54.457 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:08:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:08:54.458 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:54.471 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:08:54.480 INFO:teuthology.orchestra.run.smithi164.stdout:inferring bluefs devices from bluestore path 2024-04-26T01:08:54.481 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:54.481 INFO:teuthology.orchestra.run.smithi164.stdout: "/var/lib/ceph/osd/ceph-3/block": { 2024-04-26T01:08:54.481 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_uuid": "e9d8f764-4951-4de4-92c7-682850852e85", 2024-04-26T01:08:54.481 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 96636764160, 2024-04-26T01:08:54.481 INFO:teuthology.orchestra.run.smithi164.stdout: "btime": "2024-04-26T01:07:10.673813+0000", 2024-04-26T01:08:54.481 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "main", 2024-04-26T01:08:54.481 INFO:teuthology.orchestra.run.smithi164.stdout: "bfm_blocks": "23592960", 2024-04-26T01:08:54.481 INFO:teuthology.orchestra.run.smithi164.stdout: "bfm_blocks_per_key": "128", 2024-04-26T01:08:54.481 INFO:teuthology.orchestra.run.smithi164.stdout: "bfm_bytes_per_block": "4096", 2024-04-26T01:08:54.481 INFO:teuthology.orchestra.run.smithi164.stdout: "bfm_size": "96636764160", 2024-04-26T01:08:54.481 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs": "1", 2024-04-26T01:08:54.482 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_fsid": "4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554", 2024-04-26T01:08:54.482 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_version_when_created": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-26T01:08:54.482 INFO:teuthology.orchestra.run.smithi164.stdout: "created_at": "2024-04-26T01:07:13.116216Z", 2024-04-26T01:08:54.482 INFO:teuthology.orchestra.run.smithi164.stdout: "elastic_shared_blobs": "1", 2024-04-26T01:08:54.482 INFO:teuthology.orchestra.run.smithi164.stdout: "kv_backend": "rocksdb", 2024-04-26T01:08:54.482 INFO:teuthology.orchestra.run.smithi164.stdout: "magic": "ceph osd volume v026", 2024-04-26T01:08:54.482 INFO:teuthology.orchestra.run.smithi164.stdout: "mkfs_done": "yes", 2024-04-26T01:08:54.482 INFO:teuthology.orchestra.run.smithi164.stdout: "ready": "ready", 2024-04-26T01:08:54.482 INFO:teuthology.orchestra.run.smithi164.stdout: "require_osd_release": "19", 2024-04-26T01:08:54.482 INFO:teuthology.orchestra.run.smithi164.stdout: "whoami": "3" 2024-04-26T01:08:54.482 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:54.482 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:54.483 INFO:tasks.thrashosds.thrasher:checking if target bluestore supports sharding on osd.3 2024-04-26T01:08:54.483 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-3 show-sharding 2024-04-26T01:08:54.526 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:54.526+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:54.603 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:54.603+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:54.641 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:54.641 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:08:54.641 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:08:54.641 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:08:54.641 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 32/30 ECSubWrite(tid=1, reqid=client.4521.0:25, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 55.327670349999998, 2024-04-26T01:08:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062762640000000001, 2024-04-26T01:08:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:08:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2700000000000001e-07 2024-04-26T01:08:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313923+0000", 2024-04-26T01:08:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.0530000000000002e-05 2024-04-26T01:08:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314064+0000", 2024-04-26T01:08:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000140841 2024-04-26T01:08:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314070+0000", 2024-04-26T01:08:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.976e-06 2024-04-26T01:08:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314140+0000", 2024-04-26T01:08:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9664000000000003e-05 2024-04-26T01:08:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314151+0000", 2024-04-26T01:08:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1251e-05 2024-04-26T01:08:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320065+0000", 2024-04-26T01:08:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059141990000000002 2024-04-26T01:08:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320109+0000", 2024-04-26T01:08:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3575999999999997e-05 2024-04-26T01:08:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.ds3 33/30 ECSubWrite(tid=5, reqid=client.4521.0:35, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.325729+0000", 2024-04-26T01:08:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 55.31577369, 2024-04-26T01:08:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079681170000000003, 2024-04-26T01:08:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325729+0000", 2024-04-26T01:08:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325729+0000", 2024-04-26T01:08:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325730+0000", 2024-04-26T01:08:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.78e-07 2024-04-26T01:08:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325748+0000", 2024-04-26T01:08:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8127000000000001e-05 2024-04-26T01:08:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325789+0000", 2024-04-26T01:08:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.1100000000000003e-05 2024-04-26T01:08:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325800+0000", 2024-04-26T01:08:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0256e-05 2024-04-26T01:08:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325859+0000", 2024-04-26T01:08:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9849999999999998e-05 2024-04-26T01:08:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325869+0000", 2024-04-26T01:08:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6989999999999998e-06 2024-04-26T01:08:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.333645+0000", 2024-04-26T01:08:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0077756800000000001 2024-04-26T01:08:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.333697+0000", 2024-04-26T01:08:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2426999999999998e-05 2024-04-26T01:08:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 33/30 ECSubWrite(tid=4, reqid=client.4521.0:26, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.327582+0000", 2024-04-26T01:08:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 55.313921049999998, 2024-04-26T01:08:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0074218790000000001, 2024-04-26T01:08:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327582+0000", 2024-04-26T01:08:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327582+0000", 2024-04-26T01:08:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327583+0000", 2024-04-26T01:08:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0669999999999999e-06 2024-04-26T01:08:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327596+0000", 2024-04-26T01:08:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3135e-05 2024-04-26T01:08:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327649+0000", 2024-04-26T01:08:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2704999999999998e-05 2024-04-26T01:08:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327659+0000", 2024-04-26T01:08:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.8470000000000008e-06 2024-04-26T01:08:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327733+0000", 2024-04-26T01:08:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4114000000000002e-05 2024-04-26T01:08:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327746+0000", 2024-04-26T01:08:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3176999999999999e-05 2024-04-26T01:08:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.334969+0000", 2024-04-26T01:08:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0072232160000000002 2024-04-26T01:08:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.335004+0000", 2024-04-26T01:08:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4618e-05 2024-04-26T01:08:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 50/44 ECSubWrite(tid=213, reqid=osd.4.0:213, at_version=50'36, trim_to=0'0, roll_forward_to=47'34))", 2024-04-26T01:08:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.047619+0000", 2024-04-26T01:08:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.593883869000003, 2024-04-26T01:08:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012971391000000001, 2024-04-26T01:08:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047619+0000", 2024-04-26T01:08:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047619+0000", 2024-04-26T01:08:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047620+0000", 2024-04-26T01:08:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9999999999999997e-07 2024-04-26T01:08:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047663+0000", 2024-04-26T01:08:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3254000000000003e-05 2024-04-26T01:08:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047762+0000", 2024-04-26T01:08:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9551999999999998e-05 2024-04-26T01:08:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047793+0000", 2024-04-26T01:08:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0729999999999999e-05 2024-04-26T01:08:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.048521+0000", 2024-04-26T01:08:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000727971 2024-04-26T01:08:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.048552+0000", 2024-04-26T01:08:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1375999999999999e-05 2024-04-26T01:08:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060503+0000", 2024-04-26T01:08:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011950771000000001 2024-04-26T01:08:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060590+0000", 2024-04-26T01:08:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7137000000000003e-05 2024-04-26T01:08:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 50/44 ECSubWrite(tid=217, reqid=osd.4.0:213, at_version=0'0, trim_to=0'0, roll_forward_to=50'36))", 2024-04-26T01:08:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.063223+0000", 2024-04-26T01:08:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.578279393000003, 2024-04-26T01:08:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0071347260000000001, 2024-04-26T01:08:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063223+0000", 2024-04-26T01:08:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063223+0000", 2024-04-26T01:08:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063225+0000", 2024-04-26T01:08:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.753e-06 2024-04-26T01:08:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063249+0000", 2024-04-26T01:08:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3896999999999998e-05 2024-04-26T01:08:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063327+0000", 2024-04-26T01:08:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.7693000000000003e-05 2024-04-26T01:08:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063349+0000", 2024-04-26T01:08:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2446000000000002e-05 2024-04-26T01:08:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063511+0000", 2024-04-26T01:08:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016191799999999999 2024-04-26T01:08:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063538+0000", 2024-04-26T01:08:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7237e-05 2024-04-26T01:08:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.070265+0000", 2024-04-26T01:08:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.006726526 2024-04-26T01:08:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.070358+0000", 2024-04-26T01:08:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.3256000000000003e-05 2024-04-26T01:08:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.1s4 50/44 ECSubWrite(tid=219, reqid=osd.4.0:219, at_version=50'58, trim_to=0'0, roll_forward_to=47'56))", 2024-04-26T01:08:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.066105+0000", 2024-04-26T01:08:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.575398368999998, 2024-04-26T01:08:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079801060000000007, 2024-04-26T01:08:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066105+0000", 2024-04-26T01:08:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066105+0000", 2024-04-26T01:08:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066106+0000", 2024-04-26T01:08:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.846e-06 2024-04-26T01:08:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066156+0000", 2024-04-26T01:08:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9225999999999999e-05 2024-04-26T01:08:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066238+0000", 2024-04-26T01:08:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2126999999999995e-05 2024-04-26T01:08:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066260+0000", 2024-04-26T01:08:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2408999999999998e-05 2024-04-26T01:08:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066417+0000", 2024-04-26T01:08:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000157312 2024-04-26T01:08:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066443+0000", 2024-04-26T01:08:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5797000000000001e-05 2024-04-26T01:08:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073990+0000", 2024-04-26T01:08:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0075467670000000002 2024-04-26T01:08:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.074085+0000", 2024-04-26T01:08:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.4622000000000003e-05 2024-04-26T01:08:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 50/44 ECSubWrite(tid=220, reqid=client.4521.0:491, at_version=50'38, trim_to=0'0, roll_forward_to=50'36))", 2024-04-26T01:08:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.072911+0000", 2024-04-26T01:08:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.568591847, 2024-04-26T01:08:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0076486990000000001, 2024-04-26T01:08:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.072911+0000", 2024-04-26T01:08:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.072911+0000", 2024-04-26T01:08:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.072914+0000", 2024-04-26T01:08:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5900000000000002e-06 2024-04-26T01:08:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073741+0000", 2024-04-26T01:08:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00082785699999999999 2024-04-26T01:08:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073888+0000", 2024-04-26T01:08:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014690499999999999 2024-04-26T01:08:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073915+0000", 2024-04-26T01:08:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.614e-05 2024-04-26T01:08:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.074074+0000", 2024-04-26T01:08:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015989500000000001 2024-04-26T01:08:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.074101+0000", 2024-04-26T01:08:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6693999999999999e-05 2024-04-26T01:08:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080467+0000", 2024-04-26T01:08:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0063662689999999999 2024-04-26T01:08:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080560+0000", 2024-04-26T01:08:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2349000000000006e-05 2024-04-26T01:08:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 50/44 ECSubWrite(tid=23, reqid=osd.9.0:22, at_version=0'0, trim_to=0'0, roll_forward_to=50'37))", 2024-04-26T01:08:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.079931+0000", 2024-04-26T01:08:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.561572333000001, 2024-04-26T01:08:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066276529999999998, 2024-04-26T01:08:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079931+0000", 2024-04-26T01:08:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079931+0000", 2024-04-26T01:08:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079933+0000", 2024-04-26T01:08:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6759999999999999e-06 2024-04-26T01:08:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079965+0000", 2024-04-26T01:08:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1927000000000001e-05 2024-04-26T01:08:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080073+0000", 2024-04-26T01:08:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107796 2024-04-26T01:08:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080098+0000", 2024-04-26T01:08:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4822999999999999e-05 2024-04-26T01:08:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080244+0000", 2024-04-26T01:08:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014625799999999999 2024-04-26T01:08:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080268+0000", 2024-04-26T01:08:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4128e-05 2024-04-26T01:08:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.086463+0000", 2024-04-26T01:08:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061951300000000001 2024-04-26T01:08:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.086558+0000", 2024-04-26T01:08:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.4914999999999995e-05 2024-04-26T01:08:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 53/44 ECSubWrite(tid=249, reqid=client.4521.0:539, at_version=53'46, trim_to=0'0, roll_forward_to=51'44))", 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:19.133290+0000", 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 35.508212815, 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079087370000000008, 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.133290+0000", 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.133290+0000", 2024-04-26T01:08:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.133291+0000", 2024-04-26T01:08:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.4700000000000001e-07 2024-04-26T01:08:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.134718+0000", 2024-04-26T01:08:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001426921 2024-04-26T01:08:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.134818+0000", 2024-04-26T01:08:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000100203 2024-04-26T01:08:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.134841+0000", 2024-04-26T01:08:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2730000000000001e-05 2024-04-26T01:08:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.135022+0000", 2024-04-26T01:08:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00018121500000000001 2024-04-26T01:08:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.135051+0000", 2024-04-26T01:08:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8920000000000001e-05 2024-04-26T01:08:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.141103+0000", 2024-04-26T01:08:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0060520549999999998 2024-04-26T01:08:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.141199+0000", 2024-04-26T01:08:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6045999999999995e-05 2024-04-26T01:08:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 59/44 ECSubWrite(tid=323, reqid=osd.4.0:323, at_version=59'77, trim_to=0'0, roll_forward_to=58'73))", 2024-04-26T01:08:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.160765+0000", 2024-04-26T01:08:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 29.480737796, 2024-04-26T01:08:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066099469999999997, 2024-04-26T01:08:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160765+0000", 2024-04-26T01:08:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160765+0000", 2024-04-26T01:08:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160766+0000", 2024-04-26T01:08:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.1999999999999999e-07 2024-04-26T01:08:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160804+0000", 2024-04-26T01:08:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8068999999999998e-05 2024-04-26T01:08:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161001+0000", 2024-04-26T01:08:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000197159 2024-04-26T01:08:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161025+0000", 2024-04-26T01:08:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3768999999999999e-05 2024-04-26T01:08:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.162206+0000", 2024-04-26T01:08:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001181398 2024-04-26T01:08:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.162230+0000", 2024-04-26T01:08:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.419e-05 2024-04-26T01:08:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167200+0000", 2024-04-26T01:08:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0049697099999999996 2024-04-26T01:08:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167375+0000", 2024-04-26T01:08:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017493200000000001 2024-04-26T01:08:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 59/44 ECSubWrite(tid=325, reqid=osd.4.0:323, at_version=0'0, trim_to=0'0, roll_forward_to=59'77))", 2024-04-26T01:08:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.175746+0000", 2024-04-26T01:08:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 29.465756843000001, 2024-04-26T01:08:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0064646620000000004, 2024-04-26T01:08:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175746+0000", 2024-04-26T01:08:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175746+0000", 2024-04-26T01:08:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175748+0000", 2024-04-26T01:08:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0140000000000001e-06 2024-04-26T01:08:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175783+0000", 2024-04-26T01:08:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4946000000000001e-05 2024-04-26T01:08:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175872+0000", 2024-04-26T01:08:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8722999999999997e-05 2024-04-26T01:08:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175895+0000", 2024-04-26T01:08:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3649999999999999e-05 2024-04-26T01:08:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.176058+0000", 2024-04-26T01:08:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016219499999999999 2024-04-26T01:08:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.176085+0000", 2024-04-26T01:08:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7566e-05 2024-04-26T01:08:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.182118+0000", 2024-04-26T01:08:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0060333349999999999 2024-04-26T01:08:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.182211+0000", 2024-04-26T01:08:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2232999999999996e-05 2024-04-26T01:08:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/44 ECSubWrite(tid=353, reqid=osd.4.0:353, at_version=60'83, trim_to=0'0, roll_forward_to=59'79))", 2024-04-26T01:08:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.186337+0000", 2024-04-26T01:08:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 28.455165586, 2024-04-26T01:08:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0065831279999999997, 2024-04-26T01:08:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186337+0000", 2024-04-26T01:08:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186337+0000", 2024-04-26T01:08:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186339+0000", 2024-04-26T01:08:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.08e-06 2024-04-26T01:08:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186388+0000", 2024-04-26T01:08:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9116000000000002e-05 2024-04-26T01:08:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186438+0000", 2024-04-26T01:08:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9586000000000003e-05 2024-04-26T01:08:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186463+0000", 2024-04-26T01:08:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4859999999999999e-05 2024-04-26T01:08:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186608+0000", 2024-04-26T01:08:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000145162 2024-04-26T01:08:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186632+0000", 2024-04-26T01:08:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4179999999999999e-05 2024-04-26T01:08:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.192817+0000", 2024-04-26T01:08:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061849080000000002 2024-04-26T01:08:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.192920+0000", 2024-04-26T01:08:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000103236 2024-04-26T01:08:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 60/44 ECSubWrite(tid=52, reqid=osd.9.0:52, at_version=60'62, trim_to=0'0, roll_forward_to=60'60))", 2024-04-26T01:08:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.205252+0000", 2024-04-26T01:08:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 28.436250983000001, 2024-04-26T01:08:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.006970324, 2024-04-26T01:08:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205252+0000", 2024-04-26T01:08:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205252+0000", 2024-04-26T01:08:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205254+0000", 2024-04-26T01:08:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8169999999999999e-06 2024-04-26T01:08:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205305+0000", 2024-04-26T01:08:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.1619000000000002e-05 2024-04-26T01:08:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205912+0000", 2024-04-26T01:08:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00060660599999999999 2024-04-26T01:08:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205940+0000", 2024-04-26T01:08:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7596000000000001e-05 2024-04-26T01:08:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.206168+0000", 2024-04-26T01:08:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00022865200000000001 2024-04-26T01:08:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.206205+0000", 2024-04-26T01:08:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6439000000000001e-05 2024-04-26T01:08:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.212015+0000", 2024-04-26T01:08:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0058099400000000004 2024-04-26T01:08:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.212222+0000", 2024-04-26T01:08:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000207655 2024-04-26T01:08:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 71/44 ECSubWrite(tid=73, reqid=osd.9.0:73, at_version=71'81, trim_to=0'0, roll_forward_to=70'78))", 2024-04-26T01:08:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.110054+0000", 2024-04-26T01:08:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 17.531449156000001, 2024-04-26T01:08:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0070291959999999997, 2024-04-26T01:08:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110054+0000", 2024-04-26T01:08:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110054+0000", 2024-04-26T01:08:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110055+0000", 2024-04-26T01:08:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.48e-06 2024-04-26T01:08:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110105+0000", 2024-04-26T01:08:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9332999999999997e-05 2024-04-26T01:08:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110193+0000", 2024-04-26T01:08:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8938999999999997e-05 2024-04-26T01:08:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110214+0000", 2024-04-26T01:08:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0939999999999999e-05 2024-04-26T01:08:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110505+0000", 2024-04-26T01:08:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00029090499999999999 2024-04-26T01:08:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110521+0000", 2024-04-26T01:08:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5855999999999999e-05 2024-04-26T01:08:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.117043+0000", 2024-04-26T01:08:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0065219570000000001 2024-04-26T01:08:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.117083+0000", 2024-04-26T01:08:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.9786000000000002e-05 2024-04-26T01:08:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 71/44 ECSubWrite(tid=75, reqid=client.4521.0:731, at_version=71'83, trim_to=0'0, roll_forward_to=71'79))", 2024-04-26T01:08:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.124820+0000", 2024-04-26T01:08:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 17.516682731, 2024-04-26T01:08:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0069566530000000001, 2024-04-26T01:08:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.124820+0000", 2024-04-26T01:08:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.124820+0000", 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.124822+0000", 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8649999999999999e-06 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.125176+0000", 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00035433799999999999 2024-04-26T01:08:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.125244+0000", 2024-04-26T01:08:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7162999999999994e-05 2024-04-26T01:08:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.125258+0000", 2024-04-26T01:08:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4418e-05 2024-04-26T01:08:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.125406+0000", 2024-04-26T01:08:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014758499999999999 2024-04-26T01:08:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.125435+0000", 2024-04-26T01:08:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.932e-05 2024-04-26T01:08:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.131716+0000", 2024-04-26T01:08:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062809900000000002 2024-04-26T01:08:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.131777+0000", 2024-04-26T01:08:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0974000000000003e-05 2024-04-26T01:08:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 71/44 ECSubWrite(tid=76, reqid=client.4521.0:732, at_version=71'84, trim_to=0'0, roll_forward_to=71'81))", 2024-04-26T01:08:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.127563+0000", 2024-04-26T01:08:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 17.513939449999999, 2024-04-26T01:08:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066958829999999997, 2024-04-26T01:08:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127563+0000", 2024-04-26T01:08:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127563+0000", 2024-04-26T01:08:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127565+0000", 2024-04-26T01:08:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.17e-06 2024-04-26T01:08:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127666+0000", 2024-04-26T01:08:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000100997 2024-04-26T01:08:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127710+0000", 2024-04-26T01:08:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.4496000000000002e-05 2024-04-26T01:08:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127722+0000", 2024-04-26T01:08:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2252000000000001e-05 2024-04-26T01:08:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127954+0000", 2024-04-26T01:08:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00023175299999999999 2024-04-26T01:08:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127977+0000", 2024-04-26T01:08:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3093000000000001e-05 2024-04-26T01:08:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.134210+0000", 2024-04-26T01:08:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062328790000000002 2024-04-26T01:08:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.134259+0000", 2024-04-26T01:08:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9243000000000003e-05 2024-04-26T01:08:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 77/44 ECSubWrite(tid=89, reqid=client.4521.0:870, at_version=77'92, trim_to=0'0, roll_forward_to=74'89))", 2024-04-26T01:08:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:43.346881+0000", 2024-04-26T01:08:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 11.294621617000001, 2024-04-26T01:08:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.071918344999999995, 2024-04-26T01:08:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.346881+0000", 2024-04-26T01:08:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.346881+0000", 2024-04-26T01:08:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.346883+0000", 2024-04-26T01:08:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7099999999999999e-06 2024-04-26T01:08:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.346926+0000", 2024-04-26T01:08:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.2656e-05 2024-04-26T01:08:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.417353+0000", 2024-04-26T01:08:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.070427139999999999 2024-04-26T01:08:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.417380+0000", 2024-04-26T01:08:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7586999999999999e-05 2024-04-26T01:08:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.417538+0000", 2024-04-26T01:08:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000157429 2024-04-26T01:08:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.417556+0000", 2024-04-26T01:08:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8436000000000001e-05 2024-04-26T01:08:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.418726+0000", 2024-04-26T01:08:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0011697400000000001 2024-04-26T01:08:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.418800+0000", 2024-04-26T01:08:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.3646999999999995e-05 2024-04-26T01:08:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.1s4 81/44 ECSubWrite(tid=477, reqid=client.4521.0:895, at_version=81'99, trim_to=0'0, roll_forward_to=81'97))", 2024-04-26T01:08:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:47.483962+0000", 2024-04-26T01:08:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 7.1575404059999999, 2024-04-26T01:08:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0068314680000000003, 2024-04-26T01:08:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.483962+0000", 2024-04-26T01:08:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.483962+0000", 2024-04-26T01:08:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.483964+0000", 2024-04-26T01:08:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6169999999999999e-06 2024-04-26T01:08:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.484339+0000", 2024-04-26T01:08:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000374545 2024-04-26T01:08:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.484397+0000", 2024-04-26T01:08:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.8267999999999998e-05 2024-04-26T01:08:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.484410+0000", 2024-04-26T01:08:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3061000000000001e-05 2024-04-26T01:08:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.484487+0000", 2024-04-26T01:08:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.7156999999999993e-05 2024-04-26T01:08:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.484498+0000", 2024-04-26T01:08:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0983e-05 2024-04-26T01:08:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.490751+0000", 2024-04-26T01:08:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062530249999999997 2024-04-26T01:08:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.490794+0000", 2024-04-26T01:08:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.2812000000000003e-05 2024-04-26T01:08:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.1s4 83/44 ECSubWrite(tid=496, reqid=osd.4.0:496, at_version=83'105, trim_to=0'0, roll_forward_to=81'101))", 2024-04-26T01:08:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:49.328045+0000", 2024-04-26T01:08:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 5.3134581870000002, 2024-04-26T01:08:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0074025810000000001, 2024-04-26T01:08:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.328045+0000", 2024-04-26T01:08:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.328045+0000", 2024-04-26T01:08:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.328047+0000", 2024-04-26T01:08:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2069999999999998e-06 2024-04-26T01:08:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.328097+0000", 2024-04-26T01:08:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0139999999999998e-05 2024-04-26T01:08:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.328176+0000", 2024-04-26T01:08:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.8602000000000004e-05 2024-04-26T01:08:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.328198+0000", 2024-04-26T01:08:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.281e-05 2024-04-26T01:08:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.329482+0000", 2024-04-26T01:08:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0012835069999999999 2024-04-26T01:08:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.329510+0000", 2024-04-26T01:08:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8246e-05 2024-04-26T01:08:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.335353+0000", 2024-04-26T01:08:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0058423670000000002 2024-04-26T01:08:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.335447+0000", 2024-04-26T01:08:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.4702000000000002e-05 2024-04-26T01:08:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 88/44 ECSubWrite(tid=111, reqid=client.4521.0:950, at_version=88'106, trim_to=0'0, roll_forward_to=86'104))", 2024-04-26T01:08:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:54.078820+0000", 2024-04-26T01:08:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 0.56268310899999996, 2024-04-26T01:08:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0069099620000000004, 2024-04-26T01:08:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.078820+0000", 2024-04-26T01:08:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.078820+0000", 2024-04-26T01:08:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.078821+0000", 2024-04-26T01:08:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.54e-07 2024-04-26T01:08:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.079102+0000", 2024-04-26T01:08:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00028148000000000002 2024-04-26T01:08:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.079200+0000", 2024-04-26T01:08:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7399999999999996e-05 2024-04-26T01:08:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.079225+0000", 2024-04-26T01:08:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5669000000000001e-05 2024-04-26T01:08:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.079410+0000", 2024-04-26T01:08:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00018513899999999999 2024-04-26T01:08:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.079439+0000", 2024-04-26T01:08:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8702999999999999e-05 2024-04-26T01:08:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.085637+0000", 2024-04-26T01:08:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061982740000000001 2024-04-26T01:08:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.085730+0000", 2024-04-26T01:08:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2442999999999996e-05 2024-04-26T01:08:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:54.741 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:54.743 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:08:54.743 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:54.739+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:54.843 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:54.841+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:54.911 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:08:54.912 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:54.927 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:08:54.945 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:54.945+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:55.045 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:55.043+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:55.048 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:08:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:08:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:08:55.049 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:55.058 INFO:tasks.rados.rados.0.smithi164.stdout:378: done (3 left) 2024-04-26T01:08:55.058 INFO:tasks.rados.rados.0.smithi164.stdout:379: done (2 left) 2024-04-26T01:08:55.058 INFO:tasks.rados.rados.0.smithi164.stdout:380: done (1 left) 2024-04-26T01:08:55.058 INFO:tasks.rados.rados.0.smithi164.stdout:382: done (0 left) 2024-04-26T01:08:55.058 INFO:tasks.rados.rados.0.smithi164.stdout:383: read oid 45 snap -1 2024-04-26T01:08:55.058 INFO:tasks.rados.rados.0.smithi164.stdout:383: expect (ObjNum 150 snap 27 seq_num 150) 2024-04-26T01:08:55.058 INFO:tasks.rados.rados.0.smithi164.stdout:384: copy_from oid 4 from oid 31 current snap is 30 2024-04-26T01:08:55.058 INFO:tasks.rados.rados.0.smithi164.stdout:385: snap_create 2024-04-26T01:08:55.064 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:08:55.079 INFO:tasks.rados.rados.0.smithi164.stdout:383: expect (ObjNum 150 snap 27 seq_num 150) 2024-04-26T01:08:55.146 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:55.146+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:55.206 INFO:tasks.rados.rados.0.smithi164.stdout:384: finishing copy_from to smithi16428812-4 2024-04-26T01:08:55.206 INFO:tasks.rados.rados.0.smithi164.stdout:384: got expected ENOENT (src dne) 2024-04-26T01:08:55.206 INFO:tasks.rados.rados.0.smithi164.stdout:384: finishing copy_from racing read to smithi16428812-4 2024-04-26T01:08:55.238 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:08:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:08:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:08:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:08:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:105 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:08:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 55.810424978, 2024-04-26T01:08:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017139178000000001, 2024-04-26T01:08:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 105 2024-04-26T01:08:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.240 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:08:55.240 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.240 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.240 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.240 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:08:55.240 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.240 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.240 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.240 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425522+0000", 2024-04-26T01:08:55.240 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5430000000000001e-06 2024-04-26T01:08:55.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.240 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.240 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.241 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425529+0000", 2024-04-26T01:08:55.241 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2570000000000001e-06 2024-04-26T01:08:55.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.241 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.241 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.241 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425548+0000", 2024-04-26T01:08:55.241 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8674000000000001e-05 2024-04-26T01:08:55.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.241 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.241 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.241 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425560+0000", 2024-04-26T01:08:55.241 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2683e-05 2024-04-26T01:08:55.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.241 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.242 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.242 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428152+0000", 2024-04-26T01:08:55.242 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0025912800000000001 2024-04-26T01:08:55.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.242 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.242 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:55.242 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428234+0000", 2024-04-26T01:08:55.242 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2105000000000001e-05 2024-04-26T01:08:55.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.242 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.242 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.242 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438095+0000", 2024-04-26T01:08:55.242 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0098614180000000003 2024-04-26T01:08:55.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.243 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.243 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.243 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438205+0000", 2024-04-26T01:08:55.243 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000109849 2024-04-26T01:08:55.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.243 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.243 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.243 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442659+0000", 2024-04-26T01:08:55.243 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0044543689999999997 2024-04-26T01:08:55.243 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.243 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.243 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.243 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.244 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:117 3.0s0 3:00e538ad:::smithi16428812-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:55.244 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:08:55.244 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 55.797321982, 2024-04-26T01:08:55.244 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017480893000000001, 2024-04-26T01:08:55.244 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.244 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.244 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:55.244 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.244 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.244 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 117 2024-04-26T01:08:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:08:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:08:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.246 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438624+0000", 2024-04-26T01:08:55.246 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.6000000000000003e-07 2024-04-26T01:08:55.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.246 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.246 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.246 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438628+0000", 2024-04-26T01:08:55.246 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8029999999999998e-06 2024-04-26T01:08:55.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.246 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.246 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.246 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438635+0000", 2024-04-26T01:08:55.247 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2799999999999998e-06 2024-04-26T01:08:55.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.247 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.247 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.247 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438641+0000", 2024-04-26T01:08:55.247 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5459999999999997e-06 2024-04-26T01:08:55.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.247 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.247 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.247 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442203+0000", 2024-04-26T01:08:55.248 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003561594 2024-04-26T01:08:55.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.248 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.248 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:55.248 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442287+0000", 2024-04-26T01:08:55.248 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.4443000000000004e-05 2024-04-26T01:08:55.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.248 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.248 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.248 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451430+0000", 2024-04-26T01:08:55.248 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0091427130000000002 2024-04-26T01:08:55.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.248 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.248 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.249 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451500+0000", 2024-04-26T01:08:55.249 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9342000000000001e-05 2024-04-26T01:08:55.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.249 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.249 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.249 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.456104+0000", 2024-04-26T01:08:55.249 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0046045119999999998 2024-04-26T01:08:55.249 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.249 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.249 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.249 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.249 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:153 3.0s0 3:0230bbd3:::smithi16428812-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:55.249 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:08:55.250 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 55.746598820000003, 2024-04-26T01:08:55.250 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017766607, 2024-04-26T01:08:55.250 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.250 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.250 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:55.250 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.250 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.250 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.250 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 153 2024-04-26T01:08:55.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.250 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.250 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.250 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.250 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:08:55.251 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.251 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.251 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.251 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:08:55.251 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.251 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.251 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.251 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489347+0000", 2024-04-26T01:08:55.251 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9999999999999997e-07 2024-04-26T01:08:55.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.251 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.251 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.252 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489351+0000", 2024-04-26T01:08:55.252 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7309999999999999e-06 2024-04-26T01:08:55.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.252 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.252 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.252 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489359+0000", 2024-04-26T01:08:55.252 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.6619999999999994e-06 2024-04-26T01:08:55.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.252 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.252 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.252 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489366+0000", 2024-04-26T01:08:55.252 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.798e-06 2024-04-26T01:08:55.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.252 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.253 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.253 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492454+0000", 2024-04-26T01:08:55.253 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0030882399999999999 2024-04-26T01:08:55.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.253 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.253 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:55.253 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492562+0000", 2024-04-26T01:08:55.253 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107948 2024-04-26T01:08:55.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.253 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.253 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.253 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504997+0000", 2024-04-26T01:08:55.253 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012434176 2024-04-26T01:08:55.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.253 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.254 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.254 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.505136+0000", 2024-04-26T01:08:55.254 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013994300000000001 2024-04-26T01:08:55.254 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.254 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.254 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.254 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.507113+0000", 2024-04-26T01:08:55.254 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0019764090000000002 2024-04-26T01:08:55.254 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.254 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.254 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.254 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.255 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.255 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:287 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:08:55.255 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:04.533969+0000", 2024-04-26T01:08:55.255 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 50.701975748999999, 2024-04-26T01:08:55.255 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.018210147999999999, 2024-04-26T01:08:55.255 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.255 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 287 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533969+0000", 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533969+0000", 2024-04-26T01:08:55.256 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.257 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.257 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.257 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.257 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533971+0000", 2024-04-26T01:08:55.257 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3629999999999999e-06 2024-04-26T01:08:55.257 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.257 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.257 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.257 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533977+0000", 2024-04-26T01:08:55.257 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5799999999999997e-06 2024-04-26T01:08:55.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.258 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.258 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.258 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.534001+0000", 2024-04-26T01:08:55.258 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4006999999999999e-05 2024-04-26T01:08:55.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.258 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.258 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.258 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.534016+0000", 2024-04-26T01:08:55.258 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5160000000000001e-05 2024-04-26T01:08:55.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.258 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.258 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.259 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.545071+0000", 2024-04-26T01:08:55.259 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011054938 2024-04-26T01:08:55.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.259 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.259 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:55.259 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.545349+0000", 2024-04-26T01:08:55.259 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00027734100000000001 2024-04-26T01:08:55.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.259 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.259 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.259 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.549341+0000", 2024-04-26T01:08:55.259 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0039925150000000003 2024-04-26T01:08:55.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.259 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.260 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.260 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.549639+0000", 2024-04-26T01:08:55.260 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00029756800000000001 2024-04-26T01:08:55.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.260 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.260 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.260 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.552180+0000", 2024-04-26T01:08:55.260 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.002540676 2024-04-26T01:08:55.260 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.260 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.260 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.260 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:542 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~606208] snapc e=[e,d,c,b,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-04-26T01:08:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:20.149139+0000", 2024-04-26T01:08:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 35.086806175, 2024-04-26T01:08:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016978235000000001, 2024-04-26T01:08:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 542 2024-04-26T01:08:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.262 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.149139+0000", 2024-04-26T01:08:55.262 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.262 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.262 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.262 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.262 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.149139+0000", 2024-04-26T01:08:55.262 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.262 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.262 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.262 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.262 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.149142+0000", 2024-04-26T01:08:55.262 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9639999999999999e-06 2024-04-26T01:08:55.262 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.262 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.263 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.263 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150216+0000", 2024-04-26T01:08:55.263 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001074478 2024-04-26T01:08:55.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.263 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.263 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.263 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150251+0000", 2024-04-26T01:08:55.263 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4563000000000001e-05 2024-04-26T01:08:55.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.263 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.263 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.263 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150269+0000", 2024-04-26T01:08:55.263 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.806e-05 2024-04-26T01:08:55.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.264 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.264 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.264 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150398+0000", 2024-04-26T01:08:55.264 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012864500000000001 2024-04-26T01:08:55.264 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.264 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.264 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "op must wait for map", 2024-04-26T01:08:55.264 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150410+0000", 2024-04-26T01:08:55.264 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2427e-05 2024-04-26T01:08:55.264 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.264 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.265 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.265 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.154650+0000", 2024-04-26T01:08:55.265 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0042403129999999999 2024-04-26T01:08:55.265 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.265 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.265 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.265 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.154806+0000", 2024-04-26T01:08:55.266 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015539199999999999 2024-04-26T01:08:55.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.266 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.266 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.266 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.157843+0000", 2024-04-26T01:08:55.266 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0030374519999999999 2024-04-26T01:08:55.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.266 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.266 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.266 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.165692+0000", 2024-04-26T01:08:55.266 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0078487009999999996 2024-04-26T01:08:55.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.266 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.165984+0000", 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00029256099999999999 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.166117+0000", 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013267899999999999 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.4s5 55/30 ECSubWrite(tid=39, reqid=osd.8.0:39, at_version=55'26, trim_to=0'0, roll_forward_to=52'24))", 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:21.091694+0000", 2024-04-26T01:08:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 34.144250892000002, 2024-04-26T01:08:55.268 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.49007189200000001, 2024-04-26T01:08:55.268 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.268 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.268 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:55.268 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.268 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.268 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.268 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091694+0000", 2024-04-26T01:08:55.268 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.269 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.269 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.269 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091694+0000", 2024-04-26T01:08:55.269 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.269 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.269 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.269 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091695+0000", 2024-04-26T01:08:55.269 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.0699999999999996e-07 2024-04-26T01:08:55.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.269 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.269 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.269 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091753+0000", 2024-04-26T01:08:55.270 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7516999999999998e-05 2024-04-26T01:08:55.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.270 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.270 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580278+0000", 2024-04-26T01:08:55.270 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.48852570099999998 2024-04-26T01:08:55.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.270 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.270 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580292+0000", 2024-04-26T01:08:55.270 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3671e-05 2024-04-26T01:08:55.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.270 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.271 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580396+0000", 2024-04-26T01:08:55.271 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010389400000000001 2024-04-26T01:08:55.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.271 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.271 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.271 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580413+0000", 2024-04-26T01:08:55.271 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7121000000000001e-05 2024-04-26T01:08:55.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.271 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.271 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.271 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.581731+0000", 2024-04-26T01:08:55.271 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001318294 2024-04-26T01:08:55.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.272 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.272 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.272 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.581766+0000", 2024-04-26T01:08:55.272 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4786999999999997e-05 2024-04-26T01:08:55.272 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.272 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.272 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.272 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.272 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 58/30 ECSubWriteReply(tid=65, last_complete=58'49, committed=1, applied=1))", 2024-04-26T01:08:55.272 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:24.344800+0000", 2024-04-26T01:08:55.272 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 30.891145184999999, 2024-04-26T01:08:55.272 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.33836068000000002, 2024-04-26T01:08:55.272 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.273 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.273 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:55.273 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.273 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.273 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.273 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344800+0000", 2024-04-26T01:08:55.273 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.273 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.273 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.273 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344800+0000", 2024-04-26T01:08:55.273 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.273 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.273 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.274 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344802+0000", 2024-04-26T01:08:55.274 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.593e-06 2024-04-26T01:08:55.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.274 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.274 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.274 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344812+0000", 2024-04-26T01:08:55.274 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0943e-05 2024-04-26T01:08:55.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.274 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.274 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.274 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.682937+0000", 2024-04-26T01:08:55.274 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.33812443599999997 2024-04-26T01:08:55.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.274 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.275 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.275 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.682952+0000", 2024-04-26T01:08:55.275 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5116000000000001e-05 2024-04-26T01:08:55.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.275 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.275 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.275 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.683088+0000", 2024-04-26T01:08:55.275 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000135969 2024-04-26T01:08:55.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.275 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.276 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.276 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.683161+0000", 2024-04-26T01:08:55.276 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2622999999999999e-05 2024-04-26T01:08:55.276 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.276 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.276 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.276 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.4s5 60/30 ECSubWrite(tid=58, reqid=osd.8.0:58, at_version=60'44, trim_to=0'0, roll_forward_to=59'40))", 2024-04-26T01:08:55.276 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.181832+0000", 2024-04-26T01:08:55.276 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 29.054112965000002, 2024-04-26T01:08:55.276 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.020381277999999999, 2024-04-26T01:08:55.276 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.276 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.276 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181832+0000", 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181832+0000", 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181833+0000", 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.0999999999999998e-07 2024-04-26T01:08:55.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.278 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.278 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.278 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181871+0000", 2024-04-26T01:08:55.278 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8352999999999997e-05 2024-04-26T01:08:55.278 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.278 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.278 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.278 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.182274+0000", 2024-04-26T01:08:55.278 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00040273700000000002 2024-04-26T01:08:55.278 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.279 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.279 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.279 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.182291+0000", 2024-04-26T01:08:55.279 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6949999999999999e-05 2024-04-26T01:08:55.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.279 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.279 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.279 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187997+0000", 2024-04-26T01:08:55.279 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.005706198 2024-04-26T01:08:55.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.279 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.279 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.279 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.188021+0000", 2024-04-26T01:08:55.279 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3652999999999999e-05 2024-04-26T01:08:55.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.280 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.280 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.280 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.202131+0000", 2024-04-26T01:08:55.280 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014110531000000001 2024-04-26T01:08:55.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.280 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.280 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.280 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.202213+0000", 2024-04-26T01:08:55.280 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2145999999999996e-05 2024-04-26T01:08:55.280 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.280 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.280 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.280 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.280 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:770 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~671744] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:08:55.281 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.264210+0000", 2024-04-26T01:08:55.281 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 17.971734648000002, 2024-04-26T01:08:55.281 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016944765000000001, 2024-04-26T01:08:55.281 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.281 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.281 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:55.281 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.281 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.281 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.281 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 770 2024-04-26T01:08:55.281 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.281 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.281 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.281 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.282 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.264210+0000", 2024-04-26T01:08:55.282 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.282 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.282 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.282 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.282 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.264210+0000", 2024-04-26T01:08:55.282 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.282 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.282 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.282 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.282 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.264214+0000", 2024-04-26T01:08:55.282 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.2799999999999999e-06 2024-04-26T01:08:55.282 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.282 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.283 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.283 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.265140+0000", 2024-04-26T01:08:55.283 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00092632999999999995 2024-04-26T01:08:55.283 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.283 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.283 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.283 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.265176+0000", 2024-04-26T01:08:55.283 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6253e-05 2024-04-26T01:08:55.283 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.283 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.283 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.283 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.265192+0000", 2024-04-26T01:08:55.283 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5792999999999999e-05 2024-04-26T01:08:55.284 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.284 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.284 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.284 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.265340+0000", 2024-04-26T01:08:55.284 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000148212 2024-04-26T01:08:55.284 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.284 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.284 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.284 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.265514+0000", 2024-04-26T01:08:55.284 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017378200000000001 2024-04-26T01:08:55.284 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.284 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.284 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.284 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.267348+0000", 2024-04-26T01:08:55.284 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0018339319999999999 2024-04-26T01:08:55.285 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.285 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.285 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.285 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.280762+0000", 2024-04-26T01:08:55.285 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.01341373 2024-04-26T01:08:55.285 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.285 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.285 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:55.285 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.280956+0000", 2024-04-26T01:08:55.285 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019426400000000001 2024-04-26T01:08:55.285 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.286 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.286 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.286 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.281155+0000", 2024-04-26T01:08:55.286 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019918900000000001 2024-04-26T01:08:55.286 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.286 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.286 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.286 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.286 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:774 3.0s0 3:0982979f:::smithi16428812-35:head [setxattr _header (57) in=7b] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:08:55.286 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.273352+0000", 2024-04-26T01:08:55.287 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 17.962593104, 2024-04-26T01:08:55.287 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.31518492300000001, 2024-04-26T01:08:55.287 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.287 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.287 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:55.287 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.287 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.287 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.287 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 774 2024-04-26T01:08:55.287 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.287 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.288 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.288 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.288 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273352+0000", 2024-04-26T01:08:55.288 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.288 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.288 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.288 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273352+0000", 2024-04-26T01:08:55.288 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.289 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.289 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.289 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273353+0000", 2024-04-26T01:08:55.289 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.407e-06 2024-04-26T01:08:55.289 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.289 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.289 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.289 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273361+0000", 2024-04-26T01:08:55.289 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.8960000000000003e-06 2024-04-26T01:08:55.289 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.289 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.289 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.289 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273382+0000", 2024-04-26T01:08:55.289 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0564000000000002e-05 2024-04-26T01:08:55.289 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.290 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.290 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.290 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273397+0000", 2024-04-26T01:08:55.290 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5492999999999999e-05 2024-04-26T01:08:55.290 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.290 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.290 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.290 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.282682+0000", 2024-04-26T01:08:55.290 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0092849830000000001 2024-04-26T01:08:55.290 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.290 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.290 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.290 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.282844+0000", 2024-04-26T01:08:55.290 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000161562 2024-04-26T01:08:55.291 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.291 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.291 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.291 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.283651+0000", 2024-04-26T01:08:55.291 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00080674999999999996 2024-04-26T01:08:55.291 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.291 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.291 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.291 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.285414+0000", 2024-04-26T01:08:55.291 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0017628800000000001 2024-04-26T01:08:55.291 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.291 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.291 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:55.291 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.588422+0000", 2024-04-26T01:08:55.291 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.30300855599999998 2024-04-26T01:08:55.292 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.292 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.292 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.292 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.588537+0000", 2024-04-26T01:08:55.292 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000114832 2024-04-26T01:08:55.292 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.292 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.292 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.292 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.292 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.292 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:775 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:08:55.292 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.273465+0000", 2024-04-26T01:08:55.292 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 17.962480585000002, 2024-04-26T01:08:55.292 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.31904869699999999, 2024-04-26T01:08:55.292 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.293 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.293 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:55.293 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.293 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.293 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.293 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 775 2024-04-26T01:08:55.293 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.293 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.293 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.293 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.293 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273465+0000", 2024-04-26T01:08:55.293 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.293 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.293 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.293 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.294 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273465+0000", 2024-04-26T01:08:55.294 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.294 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.294 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.294 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273466+0000", 2024-04-26T01:08:55.294 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4929999999999999e-06 2024-04-26T01:08:55.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.294 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.294 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.294 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273472+0000", 2024-04-26T01:08:55.294 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.2040000000000002e-06 2024-04-26T01:08:55.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.294 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.295 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.295 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273487+0000", 2024-04-26T01:08:55.295 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4486e-05 2024-04-26T01:08:55.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.295 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.295 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.295 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273497+0000", 2024-04-26T01:08:55.295 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0020000000000001e-05 2024-04-26T01:08:55.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.295 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.295 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.295 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.284233+0000", 2024-04-26T01:08:55.295 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010736104999999999 2024-04-26T01:08:55.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.295 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.296 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:55.296 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.284468+0000", 2024-04-26T01:08:55.296 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000235508 2024-04-26T01:08:55.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.296 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.296 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.296 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.589239+0000", 2024-04-26T01:08:55.296 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.30477114199999999 2024-04-26T01:08:55.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.297 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.297 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.297 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.589355+0000", 2024-04-26T01:08:55.297 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000115987 2024-04-26T01:08:55.297 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.297 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.297 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.297 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.592513+0000", 2024-04-26T01:08:55.297 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0031577520000000002 2024-04-26T01:08:55.297 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.297 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.297 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.297 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.297 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.298 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:797 3.0s0 3:0d858d01:::smithi16428812-41:head [append 0~720896] snapc 16=[16,15] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-26T01:08:55.298 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:39.267373+0000", 2024-04-26T01:08:55.298 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.968571932, 2024-04-26T01:08:55.298 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.019993444999999999, 2024-04-26T01:08:55.298 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.298 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.298 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:55.298 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.298 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.298 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.298 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 797 2024-04-26T01:08:55.299 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.299 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.299 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.299 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.299 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.267373+0000", 2024-04-26T01:08:55.299 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.299 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.299 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.299 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.299 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.267373+0000", 2024-04-26T01:08:55.299 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.299 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.299 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.299 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.299 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.267377+0000", 2024-04-26T01:08:55.300 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7869999999999999e-06 2024-04-26T01:08:55.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.300 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.300 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.300 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.269934+0000", 2024-04-26T01:08:55.300 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0025572030000000001 2024-04-26T01:08:55.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.300 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.300 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.300 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.269981+0000", 2024-04-26T01:08:55.300 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.6409999999999998e-05 2024-04-26T01:08:55.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.300 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.301 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.301 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.270004+0000", 2024-04-26T01:08:55.301 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3563000000000001e-05 2024-04-26T01:08:55.301 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.301 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.301 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.301 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.270160+0000", 2024-04-26T01:08:55.301 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015629899999999999 2024-04-26T01:08:55.301 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.301 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.301 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.301 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.270398+0000", 2024-04-26T01:08:55.301 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000237915 2024-04-26T01:08:55.301 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.302 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.302 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.302 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.273343+0000", 2024-04-26T01:08:55.302 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0029442240000000001 2024-04-26T01:08:55.302 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.302 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.302 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.302 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.281421+0000", 2024-04-26T01:08:55.302 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0080788019999999995 2024-04-26T01:08:55.302 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.302 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.302 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:55.302 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.287233+0000", 2024-04-26T01:08:55.302 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0058111930000000001 2024-04-26T01:08:55.303 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.303 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.303 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.303 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.287367+0000", 2024-04-26T01:08:55.303 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013404899999999999 2024-04-26T01:08:55.303 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.303 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.303 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.303 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.303 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.303 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:801 3.0s0 3:0d858d01:::smithi16428812-41:head [setxattr _header (57) in=7b] snapc 16=[16,15] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-26T01:08:55.303 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:39.282568+0000", 2024-04-26T01:08:55.303 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.953377515, 2024-04-26T01:08:55.303 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.018474469, 2024-04-26T01:08:55.304 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.304 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.304 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:55.304 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.304 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.304 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.304 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 801 2024-04-26T01:08:55.304 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.304 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.304 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.304 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.304 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282568+0000", 2024-04-26T01:08:55.304 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.304 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.304 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.305 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.305 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282568+0000", 2024-04-26T01:08:55.305 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.305 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.305 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.305 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.305 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282569+0000", 2024-04-26T01:08:55.305 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7719999999999999e-06 2024-04-26T01:08:55.305 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.305 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.305 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.305 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282580+0000", 2024-04-26T01:08:55.305 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0562e-05 2024-04-26T01:08:55.305 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.306 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.306 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.306 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282598+0000", 2024-04-26T01:08:55.306 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7960000000000001e-05 2024-04-26T01:08:55.306 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.306 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.306 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.306 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282612+0000", 2024-04-26T01:08:55.306 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3614e-05 2024-04-26T01:08:55.306 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.306 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.306 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.306 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.289835+0000", 2024-04-26T01:08:55.306 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0072236469999999997 2024-04-26T01:08:55.307 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.307 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.307 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.307 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.290022+0000", 2024-04-26T01:08:55.307 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00018657600000000001 2024-04-26T01:08:55.307 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.307 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.307 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.307 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.290813+0000", 2024-04-26T01:08:55.307 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00079124600000000001 2024-04-26T01:08:55.307 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.307 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.307 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.307 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.292480+0000", 2024-04-26T01:08:55.307 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001667237 2024-04-26T01:08:55.308 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.308 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.308 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:55.308 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.300923+0000", 2024-04-26T01:08:55.308 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0084426929999999994 2024-04-26T01:08:55.308 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.308 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.308 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.308 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.301042+0000", 2024-04-26T01:08:55.308 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000119162 2024-04-26T01:08:55.308 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.308 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.308 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.308 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.308 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.309 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:802 3.0s0 3:0d858d01:::smithi16428812-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-04-26T01:08:55.309 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:39.282634+0000", 2024-04-26T01:08:55.309 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.953311534999999, 2024-04-26T01:08:55.309 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.025775884999999998, 2024-04-26T01:08:55.309 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.309 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.309 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:55.309 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.309 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.309 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.309 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 802 2024-04-26T01:08:55.309 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.309 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.309 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.310 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.310 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282634+0000", 2024-04-26T01:08:55.310 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.310 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.310 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.310 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282634+0000", 2024-04-26T01:08:55.310 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.310 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.310 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.310 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282635+0000", 2024-04-26T01:08:55.310 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2619999999999999e-06 2024-04-26T01:08:55.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.311 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.311 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282641+0000", 2024-04-26T01:08:55.311 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.2319999999999998e-06 2024-04-26T01:08:55.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.311 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.311 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282658+0000", 2024-04-26T01:08:55.311 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7274000000000001e-05 2024-04-26T01:08:55.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.311 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.311 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282670+0000", 2024-04-26T01:08:55.311 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1243000000000001e-05 2024-04-26T01:08:55.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.312 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.312 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.312 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.292820+0000", 2024-04-26T01:08:55.312 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010150899999999999 2024-04-26T01:08:55.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.312 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.312 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:55.312 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.293017+0000", 2024-04-26T01:08:55.312 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019624200000000001 2024-04-26T01:08:55.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.312 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.312 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.312 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.301690+0000", 2024-04-26T01:08:55.313 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0086736279999999992 2024-04-26T01:08:55.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.313 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.313 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.313 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.301945+0000", 2024-04-26T01:08:55.313 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000254885 2024-04-26T01:08:55.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.313 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.313 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.313 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.308409+0000", 2024-04-26T01:08:55.313 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0064642190000000002 2024-04-26T01:08:55.313 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.313 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.313 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.314 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.314 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:889 3.0s0 3:0d858d01:::smithi16428812-41:18 [read 0~5275648,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:08:55.314 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:47.478482+0000", 2024-04-26T01:08:55.314 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 7.7574632970000001, 2024-04-26T01:08:55.314 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.018091233000000002, 2024-04-26T01:08:55.314 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.314 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.314 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:55.314 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.314 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.314 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.314 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 889 2024-04-26T01:08:55.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.315 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.315 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.315 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.315 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478482+0000", 2024-04-26T01:08:55.315 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.315 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.315 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.315 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.315 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478482+0000", 2024-04-26T01:08:55.315 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.315 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.315 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.315 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.315 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478483+0000", 2024-04-26T01:08:55.315 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5239999999999999e-06 2024-04-26T01:08:55.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.316 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.316 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.316 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478493+0000", 2024-04-26T01:08:55.316 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7389999999999994e-06 2024-04-26T01:08:55.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.316 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.316 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.316 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478510+0000", 2024-04-26T01:08:55.316 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6549999999999999e-05 2024-04-26T01:08:55.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.316 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.316 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.316 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478519+0000", 2024-04-26T01:08:55.316 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.4450000000000002e-06 2024-04-26T01:08:55.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.317 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.317 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.317 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478604+0000", 2024-04-26T01:08:55.317 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.4685000000000007e-05 2024-04-26T01:08:55.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.317 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.317 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.317 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478738+0000", 2024-04-26T01:08:55.317 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000133967 2024-04-26T01:08:55.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.317 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.317 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.317 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.496573+0000", 2024-04-26T01:08:55.318 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017835323 2024-04-26T01:08:55.318 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.318 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.318 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.318 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.318 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:890 3.0s0 3:0d858d01:::smithi16428812-41:18 [read 0~5275648,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:08:55.318 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:47.478539+0000", 2024-04-26T01:08:55.318 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 7.7574058700000004, 2024-04-26T01:08:55.318 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.021820148000000001, 2024-04-26T01:08:55.318 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.318 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.318 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:55.318 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.318 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.319 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.319 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 890 2024-04-26T01:08:55.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.319 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.319 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.319 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478539+0000", 2024-04-26T01:08:55.319 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.319 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.319 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478539+0000", 2024-04-26T01:08:55.319 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.320 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.320 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.320 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478540+0000", 2024-04-26T01:08:55.320 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3799999999999998e-07 2024-04-26T01:08:55.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.320 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.320 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.320 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478542+0000", 2024-04-26T01:08:55.320 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8090000000000001e-06 2024-04-26T01:08:55.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.320 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.320 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.320 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478548+0000", 2024-04-26T01:08:55.320 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.04e-06 2024-04-26T01:08:55.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478551+0000", 2024-04-26T01:08:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8700000000000002e-06 2024-04-26T01:08:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.479336+0000", 2024-04-26T01:08:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00078435199999999999 2024-04-26T01:08:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.479428+0000", 2024-04-26T01:08:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2052000000000006e-05 2024-04-26T01:08:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.500359+0000", 2024-04-26T01:08:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.020931587000000001 2024-04-26T01:08:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:891 3.0s0 3:0d858d01:::smithi16428812-41:18 [read 0~5275648,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:08:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:47.478584+0000", 2024-04-26T01:08:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 7.7573608959999998, 2024-04-26T01:08:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.025384917, 2024-04-26T01:08:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:08:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 891 2024-04-26T01:08:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478584+0000", 2024-04-26T01:08:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.324 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478584+0000", 2024-04-26T01:08:55.324 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.324 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.324 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.324 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478585+0000", 2024-04-26T01:08:55.324 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3500000000000002e-07 2024-04-26T01:08:55.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.324 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.324 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.324 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478587+0000", 2024-04-26T01:08:55.324 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5859999999999999e-06 2024-04-26T01:08:55.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.324 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.324 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.325 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478594+0000", 2024-04-26T01:08:55.325 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.4799999999999998e-06 2024-04-26T01:08:55.325 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.325 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.325 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.325 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478598+0000", 2024-04-26T01:08:55.325 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.07e-06 2024-04-26T01:08:55.325 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.325 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.325 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.325 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.479740+0000", 2024-04-26T01:08:55.325 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0011423010000000001 2024-04-26T01:08:55.325 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.325 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.326 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.326 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.480024+0000", 2024-04-26T01:08:55.326 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00028346399999999998 2024-04-26T01:08:55.326 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.326 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.326 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.326 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.503969+0000", 2024-04-26T01:08:55.326 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.023945581000000001 2024-04-26T01:08:55.326 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.326 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.326 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.326 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.326 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.326 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:947 3.0s0 3:0d858d01:::smithi16428812-41:head [append 0~0] snapc 1e=[1d,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-26T01:08:55.327 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:54.058284+0000", 2024-04-26T01:08:55.327 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 1.1776611340000001, 2024-04-26T01:08:55.327 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017367719, 2024-04-26T01:08:55.327 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.327 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.327 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:55.327 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.327 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.327 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.327 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 947 2024-04-26T01:08:55.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.327 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.327 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.328 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.328 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058284+0000", 2024-04-26T01:08:55.328 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.328 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.328 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.328 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058284+0000", 2024-04-26T01:08:55.328 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.328 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.328 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.328 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058288+0000", 2024-04-26T01:08:55.328 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6440000000000002e-06 2024-04-26T01:08:55.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.328 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.329 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.329 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058303+0000", 2024-04-26T01:08:55.329 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5056e-05 2024-04-26T01:08:55.329 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.329 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.329 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.329 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058353+0000", 2024-04-26T01:08:55.329 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9855999999999998e-05 2024-04-26T01:08:55.329 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.329 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.329 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.329 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058378+0000", 2024-04-26T01:08:55.329 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5009999999999999e-05 2024-04-26T01:08:55.329 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.330 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.330 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.330 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058611+0000", 2024-04-26T01:08:55.330 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00023349500000000001 2024-04-26T01:08:55.330 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.330 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.330 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "op must wait for map", 2024-04-26T01:08:55.330 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058634+0000", 2024-04-26T01:08:55.330 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3226000000000001e-05 2024-04-26T01:08:55.330 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.330 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.330 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.330 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.061949+0000", 2024-04-26T01:08:55.330 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0033148560000000001 2024-04-26T01:08:55.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.331 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.331 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.331 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.062071+0000", 2024-04-26T01:08:55.331 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000121414 2024-04-26T01:08:55.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.331 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.331 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.331 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.062971+0000", 2024-04-26T01:08:55.331 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00090063100000000004 2024-04-26T01:08:55.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.331 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.331 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.331 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.070859+0000", 2024-04-26T01:08:55.332 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0078880219999999997 2024-04-26T01:08:55.332 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.332 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.332 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:55.332 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.075455+0000", 2024-04-26T01:08:55.332 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0045962219999999996 2024-04-26T01:08:55.332 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.332 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.332 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.332 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.075652+0000", 2024-04-26T01:08:55.332 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019628700000000001 2024-04-26T01:08:55.332 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.332 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.332 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.333 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.333 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:948 3.0s0 3:0d858d01:::smithi16428812-41:head [rollback 1d] snapc 1e=[1d,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-26T01:08:55.333 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:54.058432+0000", 2024-04-26T01:08:55.333 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 1.1775132479999999, 2024-04-26T01:08:55.333 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.020479824000000001, 2024-04-26T01:08:55.333 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.333 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.333 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:55.333 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.333 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.333 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.333 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 948 2024-04-26T01:08:55.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.333 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058432+0000", 2024-04-26T01:08:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058432+0000", 2024-04-26T01:08:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058433+0000", 2024-04-26T01:08:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.42e-06 2024-04-26T01:08:55.335 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.335 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.335 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.335 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058441+0000", 2024-04-26T01:08:55.335 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5100000000000001e-06 2024-04-26T01:08:55.335 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.335 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.335 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.335 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058492+0000", 2024-04-26T01:08:55.335 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0896000000000002e-05 2024-04-26T01:08:55.335 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.335 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.335 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.335 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058509+0000", 2024-04-26T01:08:55.335 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7119999999999999e-05 2024-04-26T01:08:55.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.336 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.336 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.336 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058774+0000", 2024-04-26T01:08:55.336 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00026504099999999999 2024-04-26T01:08:55.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.336 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.336 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:55.336 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058781+0000", 2024-04-26T01:08:55.336 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.1459999999999999e-06 2024-04-26T01:08:55.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.336 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.336 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.336 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.064161+0000", 2024-04-26T01:08:55.337 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0053797519999999998 2024-04-26T01:08:55.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.337 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.337 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.064254+0000", 2024-04-26T01:08:55.337 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.3033000000000002e-05 2024-04-26T01:08:55.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.337 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.337 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.064869+0000", 2024-04-26T01:08:55.337 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00061545899999999995 2024-04-26T01:08:55.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.337 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.337 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.072862+0000", 2024-04-26T01:08:55.338 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079924909999999991 2024-04-26T01:08:55.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.338 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.338 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:55.338 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.078761+0000", 2024-04-26T01:08:55.338 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0058996070000000003 2024-04-26T01:08:55.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.338 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.338 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.338 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.078912+0000", 2024-04-26T01:08:55.338 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015034900000000001 2024-04-26T01:08:55.338 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.338 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.339 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.339 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.339 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:949 3.0s0 3:0d858d01:::smithi16428812-41:head [append 0~0] snapc 1e=[1d,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-26T01:08:55.339 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:54.058533+0000", 2024-04-26T01:08:55.339 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 1.1774123190000001, 2024-04-26T01:08:55.339 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.020849146999999998, 2024-04-26T01:08:55.339 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:08:55.339 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:08:55.339 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:55.339 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:08:55.339 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:08:55.339 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.339 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 949 2024-04-26T01:08:55.340 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.340 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:08:55.340 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.340 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:08:55.340 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058533+0000", 2024-04-26T01:08:55.340 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.340 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.340 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.340 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:08:55.340 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058533+0000", 2024-04-26T01:08:55.340 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:08:55.340 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.340 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.340 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:08:55.340 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058534+0000", 2024-04-26T01:08:55.341 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.1999999999999998e-07 2024-04-26T01:08:55.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.341 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.341 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:08:55.341 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058538+0000", 2024-04-26T01:08:55.341 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.6870000000000002e-06 2024-04-26T01:08:55.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.341 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.341 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:08:55.341 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058549+0000", 2024-04-26T01:08:55.341 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.048e-05 2024-04-26T01:08:55.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.341 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.341 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.341 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058557+0000", 2024-04-26T01:08:55.342 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.7000000000000008e-06 2024-04-26T01:08:55.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.342 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.342 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.342 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058873+0000", 2024-04-26T01:08:55.342 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00031644699999999997 2024-04-26T01:08:55.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.342 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.342 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:55.342 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058892+0000", 2024-04-26T01:08:55.342 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.927e-05 2024-04-26T01:08:55.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.342 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.342 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:08:55.342 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.068031+0000", 2024-04-26T01:08:55.343 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0091391370000000003 2024-04-26T01:08:55.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.343 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.343 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:08:55.343 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.068307+0000", 2024-04-26T01:08:55.343 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00027563400000000002 2024-04-26T01:08:55.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.343 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.343 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.343 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.069597+0000", 2024-04-26T01:08:55.343 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001289803 2024-04-26T01:08:55.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.343 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.343 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.343 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.075671+0000", 2024-04-26T01:08:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0060739000000000001 2024-04-26T01:08:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:08:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.079183+0000", 2024-04-26T01:08:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0035121179999999998 2024-04-26T01:08:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:08:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:08:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:08:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.079382+0000", 2024-04-26T01:08:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000199051 2024-04-26T01:08:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:08:55.345 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:08:55.345 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:08:55.346 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:08:55.347 INFO:teuthology.orchestra.run.smithi164.stdout:m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 2024-04-26T01:08:55.347 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:55.344+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:55.353 INFO:tasks.thrashosds.thrasher:applying new sharding to bluestore on osd.3 2024-04-26T01:08:55.353 INFO:tasks.thrashosds.thrasher:generate random sharding 2024-04-26T01:08:55.353 INFO:tasks.thrashosds.thrasher:applying new sharding: m(1,5-) O(2) P(4,5-35) L(2,8-) 2024-04-26T01:08:55.353 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-3 --sharding 'm(1,5-) O(2) P(4,5-35) L(2,8-)' reshard 2024-04-26T01:08:55.447 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:55.444+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:08:55.476 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:55.476 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:55.476 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:08:55.476 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:55.490 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:08:55.548 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:55.547+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:55.605 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:55.606 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:55.606 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2024-04-26T01:08:55.606 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2024-04-26T01:08:55.606 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:55.620 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:08:55.649 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:55.645+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:55.658 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:08:55.750 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:55.750 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2024-04-26T01:08:55.750 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2024-04-26T01:08:55.750 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2024-04-26T01:08:55.750 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.750 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:4 3.ds0 3:bcd9cb81:::smithi16428812-2:head [append 0~32768,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:55.750 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.302987+0000", 2024-04-26T01:08:55.750 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.446534448000001, 2024-04-26T01:08:55.750 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012685460000000001, 2024-04-26T01:08:55.750 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.750 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.751 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:55.751 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:55.751 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:55.751 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.751 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 4 2024-04-26T01:08:55.751 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.751 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.751 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.751 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.751 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.302987+0000", 2024-04-26T01:08:55.751 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.751 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.751 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.751 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.752 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.302987+0000", 2024-04-26T01:08:55.752 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.752 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.752 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.752 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.752 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.302989+0000", 2024-04-26T01:08:55.752 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.266e-06 2024-04-26T01:08:55.752 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.752 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.752 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.752 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303040+0000", 2024-04-26T01:08:55.752 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1029000000000002e-05 2024-04-26T01:08:55.752 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.752 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.753 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.753 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303059+0000", 2024-04-26T01:08:55.753 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8909000000000001e-05 2024-04-26T01:08:55.753 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.753 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.753 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.753 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303074+0000", 2024-04-26T01:08:55.753 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4812e-05 2024-04-26T01:08:55.753 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.753 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.753 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.753 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303197+0000", 2024-04-26T01:08:55.753 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000122748 2024-04-26T01:08:55.753 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.754 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.754 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:55.754 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303424+0000", 2024-04-26T01:08:55.754 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022748500000000001 2024-04-26T01:08:55.754 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.754 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.754 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.754 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304556+0000", 2024-04-26T01:08:55.754 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001131948 2024-04-26T01:08:55.754 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.754 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.754 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.754 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.313947+0000", 2024-04-26T01:08:55.755 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0093914069999999992 2024-04-26T01:08:55.755 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.755 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:55.755 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315465+0000", 2024-04-26T01:08:55.755 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001517673 2024-04-26T01:08:55.755 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.755 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.755 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315672+0000", 2024-04-26T01:08:55.755 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000207183 2024-04-26T01:08:55.755 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.755 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.755 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.755 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.756 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.756 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:5 3.ds0 3:bcd9cb81:::smithi16428812-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:55.756 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.303108+0000", 2024-04-26T01:08:55.756 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.446413450000001, 2024-04-26T01:08:55.756 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013204789, 2024-04-26T01:08:55.756 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.756 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.756 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:55.756 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:55.756 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:55.756 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.756 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 5 2024-04-26T01:08:55.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.756 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.757 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.757 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303108+0000", 2024-04-26T01:08:55.757 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.757 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.757 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.757 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303108+0000", 2024-04-26T01:08:55.757 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.757 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.757 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.757 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303109+0000", 2024-04-26T01:08:55.758 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3149999999999999e-06 2024-04-26T01:08:55.758 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.758 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.758 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.758 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303117+0000", 2024-04-26T01:08:55.758 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1899999999999995e-06 2024-04-26T01:08:55.758 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.758 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.758 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.758 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303131+0000", 2024-04-26T01:08:55.758 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3890000000000001e-05 2024-04-26T01:08:55.758 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.758 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.758 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.759 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303143+0000", 2024-04-26T01:08:55.759 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1773999999999999e-05 2024-04-26T01:08:55.759 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.759 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.759 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.759 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.305710+0000", 2024-04-26T01:08:55.759 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0025668930000000002 2024-04-26T01:08:55.759 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.759 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.759 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:55.759 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.305934+0000", 2024-04-26T01:08:55.759 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022386800000000001 2024-04-26T01:08:55.759 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.759 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.759 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.760 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.307007+0000", 2024-04-26T01:08:55.760 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010735930000000001 2024-04-26T01:08:55.760 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.760 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.760 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.760 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.313953+0000", 2024-04-26T01:08:55.760 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006945657 2024-04-26T01:08:55.760 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.760 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.760 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:55.760 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316089+0000", 2024-04-26T01:08:55.760 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021360509999999999 2024-04-26T01:08:55.760 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.760 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.761 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.761 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316313+0000", 2024-04-26T01:08:55.761 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000223558 2024-04-26T01:08:55.761 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.761 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.761 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.761 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.761 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.761 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:6 3.ds0 3:bcd9cb81:::smithi16428812-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:55.761 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.303162+0000", 2024-04-26T01:08:55.761 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.446359467999997, 2024-04-26T01:08:55.761 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.027157324, 2024-04-26T01:08:55.761 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.761 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.761 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:55.762 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:55.762 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:55.762 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.762 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 6 2024-04-26T01:08:55.762 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.762 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.762 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.762 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303162+0000", 2024-04-26T01:08:55.762 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.762 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.762 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.762 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303162+0000", 2024-04-26T01:08:55.763 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.763 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.763 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.763 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.763 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303163+0000", 2024-04-26T01:08:55.763 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.145e-06 2024-04-26T01:08:55.763 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.763 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.763 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.763 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303169+0000", 2024-04-26T01:08:55.763 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8150000000000002e-06 2024-04-26T01:08:55.763 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.763 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.763 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.764 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303184+0000", 2024-04-26T01:08:55.764 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5116000000000001e-05 2024-04-26T01:08:55.764 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.764 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.764 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.764 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303196+0000", 2024-04-26T01:08:55.764 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1778e-05 2024-04-26T01:08:55.764 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.764 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.764 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.764 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308497+0000", 2024-04-26T01:08:55.764 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0053009449999999996 2024-04-26T01:08:55.764 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308937+0000", 2024-04-26T01:08:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00044013100000000002 2024-04-26T01:08:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.317636+0000", 2024-04-26T01:08:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0086997570000000007 2024-04-26T01:08:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.317809+0000", 2024-04-26T01:08:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017204899999999999 2024-04-26T01:08:55.766 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.766 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.766 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.766 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.330319+0000", 2024-04-26T01:08:55.766 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012510588 2024-04-26T01:08:55.766 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.766 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.766 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.766 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.766 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.766 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:34 3.ds0 3:b1366e28:::smithi16428812-12:head [append 0~196608,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:55.766 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321139+0000", 2024-04-26T01:08:55.766 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.428382229999997, 2024-04-26T01:08:55.766 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015400954999999999, 2024-04-26T01:08:55.767 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.767 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.767 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:55.767 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:55.767 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:55.767 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.767 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 34 2024-04-26T01:08:55.767 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.767 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.767 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.767 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.767 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321139+0000", 2024-04-26T01:08:55.767 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.768 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.768 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.768 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.768 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321139+0000", 2024-04-26T01:08:55.768 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.768 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.768 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.768 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.768 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321142+0000", 2024-04-26T01:08:55.768 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8660000000000002e-06 2024-04-26T01:08:55.768 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.768 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.768 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.768 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321475+0000", 2024-04-26T01:08:55.769 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00033334599999999999 2024-04-26T01:08:55.769 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.769 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.769 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.769 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321504+0000", 2024-04-26T01:08:55.769 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8445e-05 2024-04-26T01:08:55.769 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.769 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.769 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.769 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321518+0000", 2024-04-26T01:08:55.769 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4708e-05 2024-04-26T01:08:55.769 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.769 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.770 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.770 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322532+0000", 2024-04-26T01:08:55.770 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001013246 2024-04-26T01:08:55.770 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.770 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.770 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:55.770 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322688+0000", 2024-04-26T01:08:55.770 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000156525 2024-04-26T01:08:55.770 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.770 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.770 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.770 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323904+0000", 2024-04-26T01:08:55.770 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012160160000000001 2024-04-26T01:08:55.770 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.771 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.771 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.771 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.332759+0000", 2024-04-26T01:08:55.771 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0088551199999999993 2024-04-26T01:08:55.771 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.771 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.771 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:55.771 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336395+0000", 2024-04-26T01:08:55.771 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0036352559999999999 2024-04-26T01:08:55.771 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.771 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.771 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.771 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336540+0000", 2024-04-26T01:08:55.772 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000145427 2024-04-26T01:08:55.772 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.772 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.772 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.772 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.772 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.772 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:35 3.ds0 3:b1366e28:::smithi16428812-12:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:55.772 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321544+0000", 2024-04-26T01:08:55.772 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.427977112000001, 2024-04-26T01:08:55.772 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015414796, 2024-04-26T01:08:55.772 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.772 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.772 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:55.772 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:55.773 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:55.773 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.773 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 35 2024-04-26T01:08:55.773 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.773 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.773 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.773 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.773 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321544+0000", 2024-04-26T01:08:55.773 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.773 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.773 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.773 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.773 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321544+0000", 2024-04-26T01:08:55.773 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.774 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.774 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.774 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.774 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321545+0000", 2024-04-26T01:08:55.774 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3340000000000001e-06 2024-04-26T01:08:55.774 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.774 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.774 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.774 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321553+0000", 2024-04-26T01:08:55.774 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4739999999999999e-06 2024-04-26T01:08:55.774 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.774 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.774 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.774 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321571+0000", 2024-04-26T01:08:55.775 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7688000000000001e-05 2024-04-26T01:08:55.775 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.775 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.775 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.775 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321614+0000", 2024-04-26T01:08:55.775 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3047000000000003e-05 2024-04-26T01:08:55.775 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.775 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.775 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.775 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324961+0000", 2024-04-26T01:08:55.775 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003347184 2024-04-26T01:08:55.775 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.775 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.775 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:55.775 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.325113+0000", 2024-04-26T01:08:55.776 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000152287 2024-04-26T01:08:55.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.776 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.776 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.776 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.327055+0000", 2024-04-26T01:08:55.776 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001942292 2024-04-26T01:08:55.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.776 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.776 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.776 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.332763+0000", 2024-04-26T01:08:55.776 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0057070819999999996 2024-04-26T01:08:55.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.776 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.776 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336824+0000", 2024-04-26T01:08:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0040615449999999997 2024-04-26T01:08:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336959+0000", 2024-04-26T01:08:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013486300000000001 2024-04-26T01:08:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:36 3.ds0 3:b1366e28:::smithi16428812-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321633+0000", 2024-04-26T01:08:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.427888275000001, 2024-04-26T01:08:55.778 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.019982366000000001, 2024-04-26T01:08:55.778 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.778 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.778 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:55.778 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:55.778 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:55.778 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.778 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 36 2024-04-26T01:08:55.778 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.778 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.778 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.778 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.778 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321633+0000", 2024-04-26T01:08:55.779 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.779 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.779 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321633+0000", 2024-04-26T01:08:55.779 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.779 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.779 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321634+0000", 2024-04-26T01:08:55.779 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.268e-06 2024-04-26T01:08:55.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.779 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.780 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321641+0000", 2024-04-26T01:08:55.780 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5100000000000004e-06 2024-04-26T01:08:55.780 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.780 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.780 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.780 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321656+0000", 2024-04-26T01:08:55.780 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5191000000000001e-05 2024-04-26T01:08:55.780 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.780 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.780 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.780 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321666+0000", 2024-04-26T01:08:55.780 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0178e-05 2024-04-26T01:08:55.780 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.780 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.780 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.781 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.333161+0000", 2024-04-26T01:08:55.781 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011494605999999999 2024-04-26T01:08:55.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.781 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.781 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:55.781 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.333311+0000", 2024-04-26T01:08:55.781 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015046100000000001 2024-04-26T01:08:55.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.781 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.781 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.781 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338017+0000", 2024-04-26T01:08:55.781 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0047055489999999998 2024-04-26T01:08:55.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.781 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.782 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:55.782 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338172+0000", 2024-04-26T01:08:55.782 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015476500000000001 2024-04-26T01:08:55.782 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.782 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.782 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341615+0000", 2024-04-26T01:08:55.782 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0034438379999999998 2024-04-26T01:08:55.782 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.782 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.782 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.782 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.782 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpReadReply(3.ds0 33/30 ECSubReadReply(tid=3, attrs_read=0))", 2024-04-26T01:08:55.783 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.322592+0000", 2024-04-26T01:08:55.783 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.426929264000002, 2024-04-26T01:08:55.783 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077557909999999997, 2024-04-26T01:08:55.783 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.783 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.783 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:55.783 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.783 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.783 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322592+0000", 2024-04-26T01:08:55.783 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.783 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.784 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322592+0000", 2024-04-26T01:08:55.784 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.784 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.784 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.784 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322594+0000", 2024-04-26T01:08:55.784 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7740000000000001e-06 2024-04-26T01:08:55.784 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.784 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.784 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322617+0000", 2024-04-26T01:08:55.784 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3074999999999998e-05 2024-04-26T01:08:55.784 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.784 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.785 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322695+0000", 2024-04-26T01:08:55.785 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7892999999999994e-05 2024-04-26T01:08:55.785 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.785 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.785 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.785 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322710+0000", 2024-04-26T01:08:55.785 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.505e-05 2024-04-26T01:08:55.785 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.785 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.785 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.785 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329473+0000", 2024-04-26T01:08:55.785 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006763682 2024-04-26T01:08:55.785 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.785 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.786 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.786 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.330348+0000", 2024-04-26T01:08:55.786 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000874317 2024-04-26T01:08:55.786 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.786 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.786 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.786 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.786 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.786 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 32/30 ECSubWrite(tid=0, reqid=client.4521.0:28, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:55.786 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.326088+0000", 2024-04-26T01:08:55.786 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.423433428999999, 2024-04-26T01:08:55.786 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0097951900000000005, 2024-04-26T01:08:55.786 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.786 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.787 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:55.787 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.787 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.787 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.787 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326088+0000", 2024-04-26T01:08:55.787 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.787 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.787 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.787 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.787 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326088+0000", 2024-04-26T01:08:55.787 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.787 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.787 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.787 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.788 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326089+0000", 2024-04-26T01:08:55.788 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5460000000000001e-06 2024-04-26T01:08:55.788 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.788 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.788 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.788 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326154+0000", 2024-04-26T01:08:55.788 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4979000000000002e-05 2024-04-26T01:08:55.788 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.788 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.788 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.788 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326251+0000", 2024-04-26T01:08:55.788 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7139999999999995e-05 2024-04-26T01:08:55.788 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.788 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.789 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.789 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326266+0000", 2024-04-26T01:08:55.789 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4355999999999999e-05 2024-04-26T01:08:55.789 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.789 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.789 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.789 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.330888+0000", 2024-04-26T01:08:55.789 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0046226310000000003 2024-04-26T01:08:55.789 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.789 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.789 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.789 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.331237+0000", 2024-04-26T01:08:55.789 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000348103 2024-04-26T01:08:55.789 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.789 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.790 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.790 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335801+0000", 2024-04-26T01:08:55.790 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0045642189999999996 2024-04-26T01:08:55.790 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.790 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.790 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.790 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335883+0000", 2024-04-26T01:08:55.790 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2216e-05 2024-04-26T01:08:55.790 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.790 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.790 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.790 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.790 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.791 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:46 3.ds0 3:b85e6c81:::smithi16428812-16:head [append 0~262144,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:55.791 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.338039+0000", 2024-04-26T01:08:55.791 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.411482341000003, 2024-04-26T01:08:55.791 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0092704039999999995, 2024-04-26T01:08:55.791 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.791 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.791 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:55.791 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:55.791 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:55.791 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.791 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 46 2024-04-26T01:08:55.791 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.791 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.791 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.792 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.792 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338039+0000", 2024-04-26T01:08:55.792 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.792 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.792 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.792 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.792 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338039+0000", 2024-04-26T01:08:55.792 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.792 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.792 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.792 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.792 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338041+0000", 2024-04-26T01:08:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4660000000000002e-06 2024-04-26T01:08:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338148+0000", 2024-04-26T01:08:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000106384 2024-04-26T01:08:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338170+0000", 2024-04-26T01:08:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2022e-05 2024-04-26T01:08:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.794 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.794 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338184+0000", 2024-04-26T01:08:55.794 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4345000000000001e-05 2024-04-26T01:08:55.794 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.794 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.794 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.794 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341687+0000", 2024-04-26T01:08:55.794 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003502631 2024-04-26T01:08:55.794 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.794 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.794 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:55.794 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341929+0000", 2024-04-26T01:08:55.794 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024248 2024-04-26T01:08:55.795 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.795 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.795 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.795 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.343657+0000", 2024-04-26T01:08:55.795 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001727302 2024-04-26T01:08:55.795 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.795 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.795 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.795 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346591+0000", 2024-04-26T01:08:55.795 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0029344420000000002 2024-04-26T01:08:55.795 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.795 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.797 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:55.797 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.347148+0000", 2024-04-26T01:08:55.797 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000556801 2024-04-26T01:08:55.797 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.798 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.798 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.798 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.347309+0000", 2024-04-26T01:08:55.798 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000161531 2024-04-26T01:08:55.798 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.798 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.798 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.798 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.799 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.799 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:47 3.ds0 3:b85e6c81:::smithi16428812-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:55.799 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.338373+0000", 2024-04-26T01:08:55.799 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.411148658999998, 2024-04-26T01:08:55.799 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012969223, 2024-04-26T01:08:55.799 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.799 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.799 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:55.799 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:55.799 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:55.799 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.799 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 47 2024-04-26T01:08:55.799 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.800 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.800 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.800 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.800 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338373+0000", 2024-04-26T01:08:55.800 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.800 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.800 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.800 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.800 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338373+0000", 2024-04-26T01:08:55.800 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.800 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.800 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.800 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.800 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338374+0000", 2024-04-26T01:08:55.800 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.782e-06 2024-04-26T01:08:55.801 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.801 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.801 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.801 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338383+0000", 2024-04-26T01:08:55.801 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.6570000000000003e-06 2024-04-26T01:08:55.801 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.801 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.801 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.801 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338402+0000", 2024-04-26T01:08:55.801 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8708000000000001e-05 2024-04-26T01:08:55.801 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.801 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.801 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.801 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338415+0000", 2024-04-26T01:08:55.802 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2976000000000001e-05 2024-04-26T01:08:55.802 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.802 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.802 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.802 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.344766+0000", 2024-04-26T01:08:55.802 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0063516110000000001 2024-04-26T01:08:55.802 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.802 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.802 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:55.802 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.344969+0000", 2024-04-26T01:08:55.802 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00020233 2024-04-26T01:08:55.802 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.802 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.803 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.803 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.345881+0000", 2024-04-26T01:08:55.803 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00091244999999999998 2024-04-26T01:08:55.803 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.803 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.803 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.803 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.347458+0000", 2024-04-26T01:08:55.803 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001576945 2024-04-26T01:08:55.803 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.803 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.803 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:55.803 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.351176+0000", 2024-04-26T01:08:55.803 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0037184449999999999 2024-04-26T01:08:55.804 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.804 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.804 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.804 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.351342+0000", 2024-04-26T01:08:55.804 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016531900000000001 2024-04-26T01:08:55.804 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.804 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.804 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.804 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.804 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.804 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:48 3.ds0 3:b85e6c81:::smithi16428812-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:55.804 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.338562+0000", 2024-04-26T01:08:55.804 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.410959128999998, 2024-04-26T01:08:55.805 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.028604862000000002, 2024-04-26T01:08:55.805 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.805 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.805 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:55.805 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:55.805 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:55.805 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.805 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 48 2024-04-26T01:08:55.805 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.805 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.805 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.805 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.805 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338562+0000", 2024-04-26T01:08:55.805 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.806 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.806 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.806 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.806 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338562+0000", 2024-04-26T01:08:55.806 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.806 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.806 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.806 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.806 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338564+0000", 2024-04-26T01:08:55.806 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5319999999999999e-06 2024-04-26T01:08:55.806 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.806 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.806 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.807 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338572+0000", 2024-04-26T01:08:55.807 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1780000000000003e-06 2024-04-26T01:08:55.807 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.807 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.807 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.807 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338587+0000", 2024-04-26T01:08:55.807 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5293000000000001e-05 2024-04-26T01:08:55.807 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.807 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.807 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.807 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338599+0000", 2024-04-26T01:08:55.807 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1968e-05 2024-04-26T01:08:55.807 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.807 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.808 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.808 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346313+0000", 2024-04-26T01:08:55.808 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077139060000000004 2024-04-26T01:08:55.808 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.808 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.808 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:55.808 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346451+0000", 2024-04-26T01:08:55.808 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013819400000000001 2024-04-26T01:08:55.808 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.808 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.808 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.808 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.352103+0000", 2024-04-26T01:08:55.808 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056514249999999999 2024-04-26T01:08:55.808 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.809 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.809 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:55.809 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.352448+0000", 2024-04-26T01:08:55.809 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034553299999999997 2024-04-26T01:08:55.809 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.809 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.809 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.809 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.367167+0000", 2024-04-26T01:08:55.809 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.014718833000000001 2024-04-26T01:08:55.809 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.809 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.809 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.809 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.809 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.810 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/30 ECSubWrite(tid=2, reqid=client.4521.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:08:55.810 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.339308+0000", 2024-04-26T01:08:55.810 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.410212985999998, 2024-04-26T01:08:55.810 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0072084560000000002, 2024-04-26T01:08:55.810 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.810 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.810 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:55.810 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.810 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.810 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.810 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339308+0000", 2024-04-26T01:08:55.810 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.810 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.811 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.811 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.811 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339308+0000", 2024-04-26T01:08:55.811 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.811 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.811 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.811 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.811 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339310+0000", 2024-04-26T01:08:55.811 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4959999999999999e-06 2024-04-26T01:08:55.811 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.811 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.811 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.811 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339328+0000", 2024-04-26T01:08:55.811 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8349999999999999e-05 2024-04-26T01:08:55.812 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.812 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.812 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.812 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339614+0000", 2024-04-26T01:08:55.812 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028623 2024-04-26T01:08:55.812 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.812 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.812 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.812 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339629+0000", 2024-04-26T01:08:55.812 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4803999999999999e-05 2024-04-26T01:08:55.812 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.812 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.812 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.812 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.340313+0000", 2024-04-26T01:08:55.813 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00068382200000000005 2024-04-26T01:08:55.813 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.813 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.813 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.813 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.340400+0000", 2024-04-26T01:08:55.813 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.7321e-05 2024-04-26T01:08:55.813 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.813 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.813 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.813 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346462+0000", 2024-04-26T01:08:55.813 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0060617939999999997 2024-04-26T01:08:55.813 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.813 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.813 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.813 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346517+0000", 2024-04-26T01:08:55.814 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4639000000000003e-05 2024-04-26T01:08:55.814 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.814 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.814 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.814 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.814 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.814 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/30 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=38'19, trim_to=0'0, roll_forward_to=37'17))", 2024-04-26T01:08:55.814 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:04.427965+0000", 2024-04-26T01:08:55.814 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 51.321556442000002, 2024-04-26T01:08:55.814 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0080739560000000002, 2024-04-26T01:08:55.814 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.814 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.814 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:55.814 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.815 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.815 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.815 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.427965+0000", 2024-04-26T01:08:55.815 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.815 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.815 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.815 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.815 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.427965+0000", 2024-04-26T01:08:55.815 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.815 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.815 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.815 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.815 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.427966+0000", 2024-04-26T01:08:55.815 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3600000000000003e-07 2024-04-26T01:08:55.816 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.816 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.816 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.816 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.428022+0000", 2024-04-26T01:08:55.816 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6805999999999998e-05 2024-04-26T01:08:55.816 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.816 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.816 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.816 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.428186+0000", 2024-04-26T01:08:55.816 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000163287 2024-04-26T01:08:55.816 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.816 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.816 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.816 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.428212+0000", 2024-04-26T01:08:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6163e-05 2024-04-26T01:08:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.430074+0000", 2024-04-26T01:08:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0018619369999999999 2024-04-26T01:08:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.430098+0000", 2024-04-26T01:08:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.393e-05 2024-04-26T01:08:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.435960+0000", 2024-04-26T01:08:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0058619930000000002 2024-04-26T01:08:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.436039+0000", 2024-04-26T01:08:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9103999999999993e-05 2024-04-26T01:08:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 60/30 ECSubWrite(tid=51, reqid=osd.15.0:51, at_version=60'45, trim_to=0'0, roll_forward_to=59'43))", 2024-04-26T01:08:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:08:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 29.577521640000001, 2024-04-26T01:08:55.819 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.15434805400000001, 2024-04-26T01:08:55.819 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.819 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.819 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:55.819 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.819 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.819 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.819 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:08:55.819 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.819 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.819 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.819 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.819 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:08:55.820 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.820 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.820 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.820 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.820 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:08:55.820 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7800000000000001e-07 2024-04-26T01:08:55.820 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.820 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.820 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.820 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172035+0000", 2024-04-26T01:08:55.820 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.5151999999999998e-05 2024-04-26T01:08:55.820 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.820 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.820 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.821 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324571+0000", 2024-04-26T01:08:55.821 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.152535484 2024-04-26T01:08:55.821 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.821 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.821 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.821 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324589+0000", 2024-04-26T01:08:55.821 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8323999999999999e-05 2024-04-26T01:08:55.821 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.821 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.821 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.821 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324739+0000", 2024-04-26T01:08:55.821 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014940099999999999 2024-04-26T01:08:55.821 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.821 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.822 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.822 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324758+0000", 2024-04-26T01:08:55.822 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9562999999999999e-05 2024-04-26T01:08:55.822 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.822 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.822 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.822 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.326281+0000", 2024-04-26T01:08:55.822 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001523349 2024-04-26T01:08:55.822 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.822 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.822 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.822 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.326348+0000", 2024-04-26T01:08:55.822 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6203000000000003e-05 2024-04-26T01:08:55.823 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.823 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.823 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.823 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.823 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.823 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.bs5 60/30 ECSubWrite(tid=349, reqid=osd.4.0:349, at_version=60'52, trim_to=0'0, roll_forward_to=59'50))", 2024-04-26T01:08:55.823 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:08:55.823 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 29.571944741999999, 2024-04-26T01:08:55.823 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.007081085, 2024-04-26T01:08:55.823 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.823 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.823 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:55.823 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.823 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.824 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.824 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:08:55.824 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.824 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.824 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.824 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.824 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:08:55.824 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.824 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.824 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.824 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.824 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:08:55.824 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1799999999999999e-07 2024-04-26T01:08:55.824 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.825 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.825 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.825 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177638+0000", 2024-04-26T01:08:55.825 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1736000000000007e-05 2024-04-26T01:08:55.825 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.825 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.825 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.825 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177710+0000", 2024-04-26T01:08:55.825 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1756999999999998e-05 2024-04-26T01:08:55.825 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.825 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.825 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.825 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177722+0000", 2024-04-26T01:08:55.826 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1516e-05 2024-04-26T01:08:55.826 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.826 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.826 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.826 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177868+0000", 2024-04-26T01:08:55.826 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000146112 2024-04-26T01:08:55.826 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.826 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.826 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.826 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177896+0000", 2024-04-26T01:08:55.826 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7943999999999999e-05 2024-04-26T01:08:55.826 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.826 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.826 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.827 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.184603+0000", 2024-04-26T01:08:55.827 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006707691 2024-04-26T01:08:55.827 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.827 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.827 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.827 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.184658+0000", 2024-04-26T01:08:55.827 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4110999999999997e-05 2024-04-26T01:08:55.827 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.827 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.827 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.827 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.827 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.827 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:780 3.ds0 3:bcd9cb81:::smithi16428812-2:head [delete] snapc 15=[15] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:08:55.827 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:38.244242+0000", 2024-04-26T01:08:55.828 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 17.505279522999999, 2024-04-26T01:08:55.828 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0084976549999999998, 2024-04-26T01:08:55.828 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.828 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.828 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:55.828 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:55.828 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:55.828 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:55.828 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 780 2024-04-26T01:08:55.828 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.828 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.828 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.828 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.828 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244242+0000", 2024-04-26T01:08:55.829 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.829 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.829 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.829 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.829 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244242+0000", 2024-04-26T01:08:55.829 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.829 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.829 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.829 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.829 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244246+0000", 2024-04-26T01:08:55.829 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8260000000000003e-06 2024-04-26T01:08:55.829 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.829 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.829 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.829 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244264+0000", 2024-04-26T01:08:55.830 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8209000000000001e-05 2024-04-26T01:08:55.830 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.830 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.830 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.830 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244294+0000", 2024-04-26T01:08:55.830 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0099000000000001e-05 2024-04-26T01:08:55.830 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.830 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.830 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.830 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244310+0000", 2024-04-26T01:08:55.830 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6336000000000001e-05 2024-04-26T01:08:55.830 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.830 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.830 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.831 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244466+0000", 2024-04-26T01:08:55.831 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015627100000000001 2024-04-26T01:08:55.831 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.831 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.831 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2024-04-26T01:08:55.831 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244486+0000", 2024-04-26T01:08:55.831 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9678999999999999e-05 2024-04-26T01:08:55.831 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.831 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.831 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.831 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.246633+0000", 2024-04-26T01:08:55.831 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021467359999999998 2024-04-26T01:08:55.831 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.831 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.832 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:55.832 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.246783+0000", 2024-04-26T01:08:55.832 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015011500000000001 2024-04-26T01:08:55.832 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.832 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.832 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.832 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.248075+0000", 2024-04-26T01:08:55.832 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012918890000000001 2024-04-26T01:08:55.832 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.832 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.832 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.832 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.249771+0000", 2024-04-26T01:08:55.832 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0016958209999999999 2024-04-26T01:08:55.832 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.832 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.833 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:55.833 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.252619+0000", 2024-04-26T01:08:55.833 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002848214 2024-04-26T01:08:55.833 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.833 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.833 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.833 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.252739+0000", 2024-04-26T01:08:55.833 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012046 2024-04-26T01:08:55.833 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.833 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.833 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.833 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.833 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.833 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.3s1 72/30 ECSubWrite(tid=240, reqid=client.4521.0:781, at_version=72'20, trim_to=0'0, roll_forward_to=63'18))", 2024-04-26T01:08:55.833 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:38.263581+0000", 2024-04-26T01:08:55.834 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 17.485939954999999, 2024-04-26T01:08:55.834 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0089752990000000008, 2024-04-26T01:08:55.834 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.834 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.834 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:55.834 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.834 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.834 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.834 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.263581+0000", 2024-04-26T01:08:55.834 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.834 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.834 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.834 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.835 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.263581+0000", 2024-04-26T01:08:55.835 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.835 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.835 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.835 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.835 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.263583+0000", 2024-04-26T01:08:55.835 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.39e-06 2024-04-26T01:08:55.835 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.835 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.835 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.835 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.263949+0000", 2024-04-26T01:08:55.835 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00036607399999999998 2024-04-26T01:08:55.835 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.835 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.836 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.836 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.264035+0000", 2024-04-26T01:08:55.836 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.6055000000000002e-05 2024-04-26T01:08:55.836 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.836 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.836 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.836 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.264059+0000", 2024-04-26T01:08:55.836 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4071999999999999e-05 2024-04-26T01:08:55.836 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.836 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.836 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.836 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.264254+0000", 2024-04-26T01:08:55.836 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019476199999999999 2024-04-26T01:08:55.836 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.837 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.837 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.837 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.264282+0000", 2024-04-26T01:08:55.837 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8768999999999998e-05 2024-04-26T01:08:55.837 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.837 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.837 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.837 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.272454+0000", 2024-04-26T01:08:55.837 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0081717030000000006 2024-04-26T01:08:55.837 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.837 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.837 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.837 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.272557+0000", 2024-04-26T01:08:55.837 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010247399999999999 2024-04-26T01:08:55.837 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.838 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.838 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.838 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.838 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.838 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 72/30 ECSubWrite(tid=56, reqid=client.4521.0:785, at_version=72'49, trim_to=0'0, roll_forward_to=72'47))", 2024-04-26T01:08:55.838 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:38.303386+0000", 2024-04-26T01:08:55.838 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 17.446135602999998, 2024-04-26T01:08:55.838 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0094813009999999993, 2024-04-26T01:08:55.838 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.838 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.839 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:55.839 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.839 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.839 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.839 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.303386+0000", 2024-04-26T01:08:55.839 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.839 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.839 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.839 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.839 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.303386+0000", 2024-04-26T01:08:55.839 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.839 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.839 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.839 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.840 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.303388+0000", 2024-04-26T01:08:55.840 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8439999999999998e-06 2024-04-26T01:08:55.840 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.840 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.840 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.840 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.304973+0000", 2024-04-26T01:08:55.840 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0015846090000000001 2024-04-26T01:08:55.840 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.840 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.840 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.840 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.305084+0000", 2024-04-26T01:08:55.840 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011120099999999999 2024-04-26T01:08:55.840 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.840 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.305116+0000", 2024-04-26T01:08:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1341999999999999e-05 2024-04-26T01:08:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.305312+0000", 2024-04-26T01:08:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000196286 2024-04-26T01:08:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.305347+0000", 2024-04-26T01:08:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.5185000000000003e-05 2024-04-26T01:08:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.312795+0000", 2024-04-26T01:08:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0074474119999999996 2024-04-26T01:08:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.312867+0000", 2024-04-26T01:08:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2422000000000006e-05 2024-04-26T01:08:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.843 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 81/30 ECSubWrite(tid=67, reqid=osd.15.0:67, at_version=81'54, trim_to=0'0, roll_forward_to=74'50))", 2024-04-26T01:08:55.843 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:47.314584+0000", 2024-04-26T01:08:55.843 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 8.4349369870000004, 2024-04-26T01:08:55.843 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0072634040000000002, 2024-04-26T01:08:55.843 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.843 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.843 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:55.843 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.843 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.843 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.843 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314584+0000", 2024-04-26T01:08:55.843 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.843 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.843 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.844 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.844 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314584+0000", 2024-04-26T01:08:55.844 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.844 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.844 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.844 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.844 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314586+0000", 2024-04-26T01:08:55.844 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2729999999999999e-06 2024-04-26T01:08:55.844 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.844 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.844 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.844 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314621+0000", 2024-04-26T01:08:55.844 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.5129999999999997e-05 2024-04-26T01:08:55.844 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.845 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.845 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.845 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314685+0000", 2024-04-26T01:08:55.845 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4212000000000001e-05 2024-04-26T01:08:55.845 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.845 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.845 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.845 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314699+0000", 2024-04-26T01:08:55.845 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4453e-05 2024-04-26T01:08:55.845 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.845 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.845 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.845 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.316889+0000", 2024-04-26T01:08:55.845 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002189954 2024-04-26T01:08:55.846 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.846 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.846 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:55.846 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.316916+0000", 2024-04-26T01:08:55.846 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6441999999999999e-05 2024-04-26T01:08:55.846 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.846 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.846 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:55.846 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.321756+0000", 2024-04-26T01:08:55.846 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0048401130000000001 2024-04-26T01:08:55.846 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.846 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.846 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.847 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.321848+0000", 2024-04-26T01:08:55.847 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1827e-05 2024-04-26T01:08:55.847 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.847 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.847 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.847 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.847 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.847 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(osd.4.21:16 3.ds0 3:b85e6c81:::smithi16428812-16:head [assert-version v15,copy-get max 8388608] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-04-26T01:08:55.847 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:48.340027+0000", 2024-04-26T01:08:55.847 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 7.4094943630000003, 2024-04-26T01:08:55.847 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0093623700000000001, 2024-04-26T01:08:55.847 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:55.847 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:55.847 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:55.848 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:55.848 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.848 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:55.848 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.340027+0000", 2024-04-26T01:08:55.848 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.848 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.848 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.848 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:55.848 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.340027+0000", 2024-04-26T01:08:55.848 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:55.848 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.848 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.848 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:55.848 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.340028+0000", 2024-04-26T01:08:55.849 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1200000000000001e-07 2024-04-26T01:08:55.849 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.849 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.849 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:55.849 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.340033+0000", 2024-04-26T01:08:55.849 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.3199999999999999e-06 2024-04-26T01:08:55.849 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.849 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.849 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:55.849 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.340046+0000", 2024-04-26T01:08:55.849 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2992e-05 2024-04-26T01:08:55.849 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.849 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.849 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:55.850 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.340054+0000", 2024-04-26T01:08:55.850 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.5020000000000003e-06 2024-04-26T01:08:55.850 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.850 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.850 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:55.850 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.341775+0000", 2024-04-26T01:08:55.850 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001721045 2024-04-26T01:08:55.850 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.850 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.850 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:55.850 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.342038+0000", 2024-04-26T01:08:55.850 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000263527 2024-04-26T01:08:55.850 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:55.850 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:55.851 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:55.851 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.349389+0000", 2024-04-26T01:08:55.851 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0073510720000000002 2024-04-26T01:08:55.851 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.851 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.851 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.851 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:55.851 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:55.851 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:55.852 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:55.748+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:55.853 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:55.853 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:55.853 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:55.848+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:55.854 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:08:55.854 INFO:teuthology.orchestra.run.smithi164.stdout:reshard success 2024-04-26T01:08:55.857 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:55.857 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:55.882 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:55.883 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:55.883 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-26T01:08:55.910 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:55.910 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:55.935 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:55.935 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:55.954 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:55.951+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:55.965 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:55.966 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:55.982 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:55.983 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:55.983 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:08:55.983 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:55.997 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:08:55.998 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:55.998 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:56.025 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:56.026 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:56.055 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:56.052+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:56.058 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:56.058 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:56.059 INFO:tasks.rados.rados.0.smithi164.stdout:383: done (2 left) 2024-04-26T01:08:56.059 INFO:tasks.rados.rados.0.smithi164.stdout:384: done (1 left) 2024-04-26T01:08:56.059 INFO:tasks.rados.rados.0.smithi164.stdout:385: done (0 left) 2024-04-26T01:08:56.059 INFO:tasks.rados.rados.0.smithi164.stdout:386: rollback oid 29 current snap is 31 2024-04-26T01:08:56.059 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 29 to 25 2024-04-26T01:08:56.059 INFO:tasks.rados.rados.0.smithi164.stdout:387: delete oid 23 current snap is 31 2024-04-26T01:08:56.072 INFO:tasks.rados.rados.0.smithi164.stdout:386: finishing rollback tid 0 to smithi16428812-29 2024-04-26T01:08:56.073 INFO:tasks.rados.rados.0.smithi164.stdout:387: done (1 left) 2024-04-26T01:08:56.073 INFO:tasks.rados.rados.0.smithi164.stdout:388: snap_create 2024-04-26T01:08:56.074 INFO:tasks.rados.rados.0.smithi164.stdout:386: finishing rollback tid 1 to smithi16428812-29 2024-04-26T01:08:56.074 INFO:tasks.rados.rados.0.smithi164.stdout:386: finishing rollback tid 2 to smithi16428812-29 2024-04-26T01:08:56.074 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 29 v 75 (ObjNum 98 snap 11 seq_num 98) dirty exists 2024-04-26T01:08:56.083 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-26T01:08:56.088 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:56.089 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:56.113 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:08:56.113 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:08:56.119 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:56.120 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:08:56.120 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2024-04-26T01:08:56.120 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2024-04-26T01:08:56.120 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:56.134 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:08:56.135 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:56.135 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:56.156 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:56.153+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:56.160 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:56.160 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:56.189 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:56.190 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:56.224 INFO:tasks.thrashosds.thrasher:running fsck to verify new sharding on osd.3 2024-04-26T01:08:56.225 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-3 fsck 2024-04-26T01:08:56.253 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:08:56.253 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2024-04-26T01:08:56.253 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2024-04-26T01:08:56.253 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2024-04-26T01:08:56.253 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.253 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:29 3.5s0 3:acf4d671:::smithi16428812-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:56.253 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321678+0000", 2024-04-26T01:08:56.253 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.929997991999997, 2024-04-26T01:08:56.253 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.016320530999999999, 2024-04-26T01:08:56.253 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.253 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.254 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:56.254 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:56.254 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:56.254 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:56.254 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 29 2024-04-26T01:08:56.254 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.254 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.254 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.254 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.254 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321678+0000", 2024-04-26T01:08:56.254 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.254 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.254 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.255 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.255 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321678+0000", 2024-04-26T01:08:56.255 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.255 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.255 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.255 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.255 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321680+0000", 2024-04-26T01:08:56.255 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1680000000000001e-06 2024-04-26T01:08:56.255 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.255 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.255 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.255 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321686+0000", 2024-04-26T01:08:56.255 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9539999999999998e-06 2024-04-26T01:08:56.255 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.256 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.256 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.256 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321702+0000", 2024-04-26T01:08:56.256 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5241e-05 2024-04-26T01:08:56.256 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.256 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.256 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.256 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321716+0000", 2024-04-26T01:08:56.256 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4384e-05 2024-04-26T01:08:56.256 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.256 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.256 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.256 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329428+0000", 2024-04-26T01:08:56.256 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077114369999999998 2024-04-26T01:08:56.257 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.257 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.257 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:56.257 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329669+0000", 2024-04-26T01:08:56.257 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000241561 2024-04-26T01:08:56.257 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.257 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.257 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:56.257 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.331103+0000", 2024-04-26T01:08:56.257 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001433871 2024-04-26T01:08:56.257 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.257 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.257 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:56.257 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336734+0000", 2024-04-26T01:08:56.258 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.005631306 2024-04-26T01:08:56.258 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.258 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.258 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:56.258 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337654+0000", 2024-04-26T01:08:56.258 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000919716 2024-04-26T01:08:56.258 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.258 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.258 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.258 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337999+0000", 2024-04-26T01:08:56.258 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034489299999999998 2024-04-26T01:08:56.258 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.258 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.258 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.259 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.259 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.259 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:30 3.5s0 3:acf4d671:::smithi16428812-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:56.259 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.322117+0000", 2024-04-26T01:08:56.259 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.929558937000003, 2024-04-26T01:08:56.259 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.021239319999999999, 2024-04-26T01:08:56.259 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.259 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.259 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:56.259 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:56.259 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:56.259 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:56.259 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 30 2024-04-26T01:08:56.259 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.260 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.260 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.260 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.260 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322117+0000", 2024-04-26T01:08:56.260 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.260 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.260 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.260 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.260 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322117+0000", 2024-04-26T01:08:56.260 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.260 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.260 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.260 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.260 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322119+0000", 2024-04-26T01:08:56.261 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3740000000000001e-06 2024-04-26T01:08:56.261 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.261 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.261 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.261 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322127+0000", 2024-04-26T01:08:56.261 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.994e-06 2024-04-26T01:08:56.261 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.261 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.261 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.261 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322146+0000", 2024-04-26T01:08:56.261 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9205999999999999e-05 2024-04-26T01:08:56.261 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.261 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.261 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.262 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322159+0000", 2024-04-26T01:08:56.262 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3377000000000001e-05 2024-04-26T01:08:56.262 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.262 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.262 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.262 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.334766+0000", 2024-04-26T01:08:56.262 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012606732000000001 2024-04-26T01:08:56.262 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.262 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.262 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:56.262 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335000+0000", 2024-04-26T01:08:56.262 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00023435799999999999 2024-04-26T01:08:56.262 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.262 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.263 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.263 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338932+0000", 2024-04-26T01:08:56.263 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0039317500000000003 2024-04-26T01:08:56.263 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.263 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.263 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:56.263 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339070+0000", 2024-04-26T01:08:56.263 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000137315 2024-04-26T01:08:56.263 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.263 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.263 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.263 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.343357+0000", 2024-04-26T01:08:56.263 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042872140000000001 2024-04-26T01:08:56.263 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.263 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.264 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.264 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.264 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.264 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:128 3.5s0 3:a661dd24:::smithi16428812-43:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:56.264 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:56.264 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.795190814000001, 2024-04-26T01:08:56.264 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.083673129999999998, 2024-04-26T01:08:56.264 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.264 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.264 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:56.264 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:56.264 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:56.264 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:56.264 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 128 2024-04-26T01:08:56.265 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.265 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.265 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.265 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.265 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:56.265 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.265 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.265 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.265 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.265 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:56.265 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.265 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.265 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.265 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.265 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:08:56.266 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1299999999999999e-07 2024-04-26T01:08:56.266 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.266 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.266 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.266 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456490+0000", 2024-04-26T01:08:56.266 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4029999999999999e-06 2024-04-26T01:08:56.266 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.266 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.266 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.266 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456501+0000", 2024-04-26T01:08:56.266 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1124999999999999e-05 2024-04-26T01:08:56.266 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.266 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.266 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.267 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456507+0000", 2024-04-26T01:08:56.267 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3180000000000003e-06 2024-04-26T01:08:56.267 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.267 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.267 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.267 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.458775+0000", 2024-04-26T01:08:56.267 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002268061 2024-04-26T01:08:56.267 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.267 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.267 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:56.267 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.458948+0000", 2024-04-26T01:08:56.267 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017263900000000001 2024-04-26T01:08:56.267 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.267 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.268 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:56.268 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.459498+0000", 2024-04-26T01:08:56.268 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00055037900000000004 2024-04-26T01:08:56.268 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.268 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.268 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:56.268 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460166+0000", 2024-04-26T01:08:56.268 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00066757899999999996 2024-04-26T01:08:56.268 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.268 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.268 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:56.268 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540012+0000", 2024-04-26T01:08:56.268 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.079846146000000007 2024-04-26T01:08:56.268 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.269 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.269 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.269 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540159+0000", 2024-04-26T01:08:56.269 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014676700000000001 2024-04-26T01:08:56.269 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.269 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.269 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.269 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.269 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.269 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:129 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:56.269 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:56.269 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.795161843000002, 2024-04-26T01:08:56.269 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.09570555, 2024-04-26T01:08:56.270 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.270 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.270 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:56.270 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:56.270 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:56.270 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:56.270 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 129 2024-04-26T01:08:56.270 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.270 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.270 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.270 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.270 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:56.270 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.270 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.271 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.271 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.271 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:56.271 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.271 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.271 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.271 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.271 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:08:56.271 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0699999999999997e-07 2024-04-26T01:08:56.271 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.271 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.271 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.271 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456518+0000", 2024-04-26T01:08:56.271 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.898e-06 2024-04-26T01:08:56.271 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.272 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.272 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.272 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456532+0000", 2024-04-26T01:08:56.272 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3597e-05 2024-04-26T01:08:56.272 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.272 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.272 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.272 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456537+0000", 2024-04-26T01:08:56.272 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.242e-06 2024-04-26T01:08:56.272 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.272 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.272 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.272 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.459839+0000", 2024-04-26T01:08:56.272 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033019830000000001 2024-04-26T01:08:56.273 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.273 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.273 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:56.273 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.459972+0000", 2024-04-26T01:08:56.273 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000133199 2024-04-26T01:08:56.273 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.273 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.273 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.273 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540208+0000", 2024-04-26T01:08:56.273 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.080236319 2024-04-26T01:08:56.273 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.273 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.273 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:56.273 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540275+0000", 2024-04-26T01:08:56.273 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6486e-05 2024-04-26T01:08:56.274 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.274 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.274 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.274 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.552220+0000", 2024-04-26T01:08:56.274 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011945318999999999 2024-04-26T01:08:56.274 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.274 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.274 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.274 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.274 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.274 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 33/30 ECSubWriteReply(tid=9, last_complete=33'6, committed=1, applied=1))", 2024-04-26T01:08:56.274 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:56.274 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.791380769, 2024-04-26T01:08:56.274 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.079879818000000005, 2024-04-26T01:08:56.275 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.275 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.275 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:56.275 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.275 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.275 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.275 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:56.275 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.275 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.275 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.275 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.275 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:56.275 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.275 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.276 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.276 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.276 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:08:56.276 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6599999999999996e-07 2024-04-26T01:08:56.276 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.276 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.276 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.276 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460301+0000", 2024-04-26T01:08:56.276 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4689999999999999e-06 2024-04-26T01:08:56.276 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.276 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.276 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.276 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.539830+0000", 2024-04-26T01:08:56.276 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.079528926999999999 2024-04-26T01:08:56.276 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.539834+0000", 2024-04-26T01:08:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5430000000000004e-06 2024-04-26T01:08:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.539887+0000", 2024-04-26T01:08:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2998999999999999e-05 2024-04-26T01:08:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540175+0000", 2024-04-26T01:08:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028821400000000002 2024-04-26T01:08:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.278 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.278 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.278 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.278 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.278 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:183 3.5s0 3:a8ed6f44:::smithi16428812-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:56.278 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:56.278 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.716482781000003, 2024-04-26T01:08:56.278 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.017578818999999999, 2024-04-26T01:08:56.278 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.278 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.278 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:56.278 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:56.278 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:56.278 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:56.279 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 183 2024-04-26T01:08:56.279 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.279 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.279 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.279 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.279 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:56.279 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.279 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.279 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.279 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.279 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:56.279 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.279 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.279 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.280 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.280 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:08:56.280 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9500000000000002e-07 2024-04-26T01:08:56.280 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.280 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.280 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.280 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535198+0000", 2024-04-26T01:08:56.280 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6430000000000001e-06 2024-04-26T01:08:56.280 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.280 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.280 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.280 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535207+0000", 2024-04-26T01:08:56.280 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9730000000000003e-06 2024-04-26T01:08:56.280 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.280 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.281 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.281 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535215+0000", 2024-04-26T01:08:56.281 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1540000000000002e-06 2024-04-26T01:08:56.281 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.281 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.281 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.281 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.537794+0000", 2024-04-26T01:08:56.281 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0025788740000000001 2024-04-26T01:08:56.281 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.281 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.281 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:56.281 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.537868+0000", 2024-04-26T01:08:56.281 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4028000000000003e-05 2024-04-26T01:08:56.281 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.282 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.282 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.282 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.550967+0000", 2024-04-26T01:08:56.282 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013099006999999999 2024-04-26T01:08:56.282 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.282 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.282 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:56.282 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551052+0000", 2024-04-26T01:08:56.282 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4667999999999996e-05 2024-04-26T01:08:56.282 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.282 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.282 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.282 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.552772+0000", 2024-04-26T01:08:56.282 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0017207769999999999 2024-04-26T01:08:56.283 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.283 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.283 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.283 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.283 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.283 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 33/30 ECSubWriteReply(tid=14, last_complete=33'9, committed=1, applied=1))", 2024-04-26T01:08:56.283 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.551733+0000", 2024-04-26T01:08:56.283 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.699943320000003, 2024-04-26T01:08:56.283 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.40949085299999999, 2024-04-26T01:08:56.283 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.283 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.283 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:56.283 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.283 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.283 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.284 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551733+0000", 2024-04-26T01:08:56.284 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.284 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.284 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.284 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.284 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551733+0000", 2024-04-26T01:08:56.284 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.284 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.284 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.284 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.284 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551734+0000", 2024-04-26T01:08:56.284 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7199999999999998e-07 2024-04-26T01:08:56.284 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.284 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.285 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.285 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551739+0000", 2024-04-26T01:08:56.285 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9520000000000002e-06 2024-04-26T01:08:56.285 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.285 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.285 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.285 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.960976+0000", 2024-04-26T01:08:56.285 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.40923691400000001 2024-04-26T01:08:56.285 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.285 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.285 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.285 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.960989+0000", 2024-04-26T01:08:56.285 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3536000000000001e-05 2024-04-26T01:08:56.285 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.286 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.286 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.286 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.961157+0000", 2024-04-26T01:08:56.286 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016794600000000001 2024-04-26T01:08:56.286 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.286 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.286 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.286 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.961224+0000", 2024-04-26T01:08:56.286 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6733e-05 2024-04-26T01:08:56.286 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.286 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.286 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.286 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.286 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.287 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:251 3.5s0 3:acf4d671:::smithi16428812-10:head [append 0~0] snapc 4=[4,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:08:56.287 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:03.428674+0000", 2024-04-26T01:08:56.287 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 52.823001883000003, 2024-04-26T01:08:56.287 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.48188827299999998, 2024-04-26T01:08:56.287 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.287 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.287 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:56.287 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:56.287 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:56.287 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:56.287 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 251 2024-04-26T01:08:56.287 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.287 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.287 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428674+0000", 2024-04-26T01:08:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428674+0000", 2024-04-26T01:08:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428676+0000", 2024-04-26T01:08:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6559999999999999e-06 2024-04-26T01:08:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428685+0000", 2024-04-26T01:08:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0869999999999996e-06 2024-04-26T01:08:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428707+0000", 2024-04-26T01:08:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1365999999999998e-05 2024-04-26T01:08:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428723+0000", 2024-04-26T01:08:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6563e-05 2024-04-26T01:08:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.434334+0000", 2024-04-26T01:08:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056104170000000004 2024-04-26T01:08:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.434677+0000", 2024-04-26T01:08:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000343937 2024-04-26T01:08:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.436169+0000", 2024-04-26T01:08:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014915939999999999 2024-04-26T01:08:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.438266+0000", 2024-04-26T01:08:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002096746 2024-04-26T01:08:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.910445+0000", 2024-04-26T01:08:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.47217926300000002 2024-04-26T01:08:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.910563+0000", 2024-04-26T01:08:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000117644 2024-04-26T01:08:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:416 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-26T01:08:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.443980+0000", 2024-04-26T01:08:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 45.807696493000002, 2024-04-26T01:08:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.018880322000000001, 2024-04-26T01:08:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 416 2024-04-26T01:08:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443980+0000", 2024-04-26T01:08:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443980+0000", 2024-04-26T01:08:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443981+0000", 2024-04-26T01:08:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7000000000000003e-07 2024-04-26T01:08:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443984+0000", 2024-04-26T01:08:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.36e-06 2024-04-26T01:08:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443994+0000", 2024-04-26T01:08:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0285999999999999e-05 2024-04-26T01:08:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.444002+0000", 2024-04-26T01:08:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4970000000000004e-06 2024-04-26T01:08:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.454900+0000", 2024-04-26T01:08:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010897784000000001 2024-04-26T01:08:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.455102+0000", 2024-04-26T01:08:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00020183000000000001 2024-04-26T01:08:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.460460+0000", 2024-04-26T01:08:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0053580140000000004 2024-04-26T01:08:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.460760+0000", 2024-04-26T01:08:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00030007900000000002 2024-04-26T01:08:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.462860+0000", 2024-04-26T01:08:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021005020000000002 2024-04-26T01:08:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.3s3 44/30 ECSubWrite(tid=198, reqid=client.4521.0:420, at_version=44'13, trim_to=0'0, roll_forward_to=44'11))", 2024-04-26T01:08:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.466337+0000", 2024-04-26T01:08:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 45.785339735999997, 2024-04-26T01:08:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.57864738100000002, 2024-04-26T01:08:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466337+0000", 2024-04-26T01:08:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466337+0000", 2024-04-26T01:08:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466338+0000", 2024-04-26T01:08:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4479999999999999e-06 2024-04-26T01:08:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466374+0000", 2024-04-26T01:08:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6137999999999999e-05 2024-04-26T01:08:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.043926+0000", 2024-04-26T01:08:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.57755188800000001 2024-04-26T01:08:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.043944+0000", 2024-04-26T01:08:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7745999999999999e-05 2024-04-26T01:08:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044049+0000", 2024-04-26T01:08:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010500700000000001 2024-04-26T01:08:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044068+0000", 2024-04-26T01:08:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8984999999999999e-05 2024-04-26T01:08:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044949+0000", 2024-04-26T01:08:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00088138400000000003 2024-04-26T01:08:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044984+0000", 2024-04-26T01:08:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4785e-05 2024-04-26T01:08:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:554 3.5s0 3:a661dd24:::smithi16428812-43:head [copy-from ver 26] snapc e=[e,d,b,7] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-04-26T01:08:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:21.094110+0000", 2024-04-26T01:08:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 35.157565947000002, 2024-04-26T01:08:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.050017671, 2024-04-26T01:08:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 554 2024-04-26T01:08:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094110+0000", 2024-04-26T01:08:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094110+0000", 2024-04-26T01:08:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.303 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.303 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.303 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.303 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094114+0000", 2024-04-26T01:08:56.303 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.89e-06 2024-04-26T01:08:56.303 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.303 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.303 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.303 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094134+0000", 2024-04-26T01:08:56.303 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0007e-05 2024-04-26T01:08:56.303 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.303 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.303 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.303 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094181+0000", 2024-04-26T01:08:56.303 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6731999999999999e-05 2024-04-26T01:08:56.304 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.304 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.304 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.304 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094205+0000", 2024-04-26T01:08:56.304 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4326999999999999e-05 2024-04-26T01:08:56.304 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.304 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.304 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.304 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094455+0000", 2024-04-26T01:08:56.304 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024929999999999999 2024-04-26T01:08:56.304 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.304 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.304 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2024-04-26T01:08:56.304 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094477+0000", 2024-04-26T01:08:56.305 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2646e-05 2024-04-26T01:08:56.305 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.305 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.305 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.305 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105141+0000", 2024-04-26T01:08:56.305 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010664167 2024-04-26T01:08:56.305 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.305 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.305 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:56.305 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105392+0000", 2024-04-26T01:08:56.305 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00025070700000000002 2024-04-26T01:08:56.305 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.305 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.305 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:56.305 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.135272+0000", 2024-04-26T01:08:56.306 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.029880263000000001 2024-04-26T01:08:56.306 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.306 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.306 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:56.306 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.142569+0000", 2024-04-26T01:08:56.306 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0072964270000000003 2024-04-26T01:08:56.306 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.306 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.306 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:56.306 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.142755+0000", 2024-04-26T01:08:56.306 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00018603399999999999 2024-04-26T01:08:56.306 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.306 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.306 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.307 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.144128+0000", 2024-04-26T01:08:56.307 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0013731710000000001 2024-04-26T01:08:56.307 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.307 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.307 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.307 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.307 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.307 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:555 3.5s0 3:a661dd24:::smithi16428812-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e55)", 2024-04-26T01:08:56.307 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:21.094258+0000", 2024-04-26T01:08:56.307 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 35.157418782000001, 2024-04-26T01:08:56.307 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.045831917999999999, 2024-04-26T01:08:56.307 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.307 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.307 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:56.307 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:56.308 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:56.308 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:56.308 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 555 2024-04-26T01:08:56.308 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.308 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.308 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.308 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.308 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094258+0000", 2024-04-26T01:08:56.308 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.308 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.308 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.308 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.308 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094258+0000", 2024-04-26T01:08:56.308 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.309 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.309 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.309 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.309 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094259+0000", 2024-04-26T01:08:56.309 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7370000000000001e-06 2024-04-26T01:08:56.309 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.309 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.309 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.309 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094268+0000", 2024-04-26T01:08:56.309 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8359999999999998e-06 2024-04-26T01:08:56.309 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.309 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.309 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.309 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094289+0000", 2024-04-26T01:08:56.309 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0937e-05 2024-04-26T01:08:56.310 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.310 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.310 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.310 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094299+0000", 2024-04-26T01:08:56.310 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.029e-05 2024-04-26T01:08:56.310 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.310 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.310 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.310 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094609+0000", 2024-04-26T01:08:56.310 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00031005900000000001 2024-04-26T01:08:56.310 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.310 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.310 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:56.310 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094619+0000", 2024-04-26T01:08:56.311 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9799999999999993e-06 2024-04-26T01:08:56.311 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.311 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.311 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.311 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105868+0000", 2024-04-26T01:08:56.311 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011248555 2024-04-26T01:08:56.311 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.311 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.311 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2024-04-26T01:08:56.311 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105946+0000", 2024-04-26T01:08:56.311 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8005999999999998e-05 2024-04-26T01:08:56.311 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.311 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.311 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.312 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.139672+0000", 2024-04-26T01:08:56.312 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.033725805999999997 2024-04-26T01:08:56.312 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.312 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.312 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:56.312 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.139869+0000", 2024-04-26T01:08:56.312 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019683 2024-04-26T01:08:56.312 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.312 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.312 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.312 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.140089+0000", 2024-04-26T01:08:56.312 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022088199999999999 2024-04-26T01:08:56.312 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.312 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.313 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.313 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.313 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.313 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s4 71/44 ECSubWrite(tid=234, reqid=client.4521.0:741, at_version=71'32, trim_to=0'0, roll_forward_to=71'28))", 2024-04-26T01:08:56.313 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:37.153424+0000", 2024-04-26T01:08:56.313 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 19.098251878999999, 2024-04-26T01:08:56.313 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.162277859, 2024-04-26T01:08:56.313 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.313 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.313 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:56.313 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.313 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.313 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.313 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.153424+0000", 2024-04-26T01:08:56.314 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.314 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.314 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.314 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.314 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.153424+0000", 2024-04-26T01:08:56.314 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.314 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.314 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.314 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.314 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.153426+0000", 2024-04-26T01:08:56.314 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8729999999999999e-06 2024-04-26T01:08:56.314 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.314 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.314 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.315 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.153470+0000", 2024-04-26T01:08:56.315 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3185000000000001e-05 2024-04-26T01:08:56.315 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.315 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.315 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.315 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.313873+0000", 2024-04-26T01:08:56.315 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.160403237 2024-04-26T01:08:56.315 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.315 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.315 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.315 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.313888+0000", 2024-04-26T01:08:56.315 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4796e-05 2024-04-26T01:08:56.315 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.315 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.315 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.316 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.314082+0000", 2024-04-26T01:08:56.316 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019490800000000001 2024-04-26T01:08:56.316 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.316 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.316 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:56.316 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.314120+0000", 2024-04-26T01:08:56.316 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7954999999999998e-05 2024-04-26T01:08:56.316 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.316 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.316 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:56.316 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.315656+0000", 2024-04-26T01:08:56.316 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0015357960000000001 2024-04-26T01:08:56.316 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.316 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.316 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.317 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.315702+0000", 2024-04-26T01:08:56.317 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6109000000000002e-05 2024-04-26T01:08:56.317 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.317 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.317 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.317 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.317 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.317 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.0s2 71/30 ECSubWrite(tid=78, reqid=client.4521.0:774, at_version=71'64, trim_to=0'0, roll_forward_to=71'61))", 2024-04-26T01:08:56.317 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:37.284626+0000", 2024-04-26T01:08:56.317 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 18.967050167, 2024-04-26T01:08:56.317 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.30429075900000002, 2024-04-26T01:08:56.317 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.317 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.317 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:56.318 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.318 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.318 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.318 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.284626+0000", 2024-04-26T01:08:56.318 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.318 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.318 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.318 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.318 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.284626+0000", 2024-04-26T01:08:56.318 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.318 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.318 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.318 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.318 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.284628+0000", 2024-04-26T01:08:56.318 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5570000000000001e-06 2024-04-26T01:08:56.319 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.319 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.319 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.319 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.284659+0000", 2024-04-26T01:08:56.319 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1751999999999997e-05 2024-04-26T01:08:56.319 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.319 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.319 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.319 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.587683+0000", 2024-04-26T01:08:56.319 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.30302400200000001 2024-04-26T01:08:56.319 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.319 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.319 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.319 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.587698+0000", 2024-04-26T01:08:56.320 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5006e-05 2024-04-26T01:08:56.320 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.320 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.320 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.320 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.587817+0000", 2024-04-26T01:08:56.320 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011894499999999999 2024-04-26T01:08:56.320 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.320 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.320 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:56.320 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.587840+0000", 2024-04-26T01:08:56.320 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2218999999999999e-05 2024-04-26T01:08:56.320 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.320 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.320 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:56.320 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.588819+0000", 2024-04-26T01:08:56.321 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00097946199999999995 2024-04-26T01:08:56.321 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.321 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.321 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.321 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.588917+0000", 2024-04-26T01:08:56.321 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7816e-05 2024-04-26T01:08:56.321 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.321 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.321 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.321 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.321 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.321 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s4 71/44 ECSubWrite(tid=238, reqid=client.4521.0:741, at_version=0'0, trim_to=0'0, roll_forward_to=71'32))", 2024-04-26T01:08:56.321 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:37.316333+0000", 2024-04-26T01:08:56.322 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 18.935343380999999, 2024-04-26T01:08:56.322 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.27762489099999998, 2024-04-26T01:08:56.322 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.322 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.322 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:08:56.322 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.322 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.322 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.322 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.316333+0000", 2024-04-26T01:08:56.322 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.322 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.322 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.322 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.322 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.316333+0000", 2024-04-26T01:08:56.323 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.323 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.323 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.323 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.323 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.316334+0000", 2024-04-26T01:08:56.323 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4169999999999999e-06 2024-04-26T01:08:56.323 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.323 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.323 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.323 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.316361+0000", 2024-04-26T01:08:56.323 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6299e-05 2024-04-26T01:08:56.323 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.323 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.323 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.324 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.592405+0000", 2024-04-26T01:08:56.324 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.27604479100000001 2024-04-26T01:08:56.324 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.324 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.324 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.324 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.592428+0000", 2024-04-26T01:08:56.324 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2039e-05 2024-04-26T01:08:56.324 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.324 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.324 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.324 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.592591+0000", 2024-04-26T01:08:56.324 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016372700000000001 2024-04-26T01:08:56.324 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.324 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.325 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:56.325 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.592613+0000", 2024-04-26T01:08:56.325 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2118999999999999e-05 2024-04-26T01:08:56.325 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.325 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.325 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:56.325 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.593876+0000", 2024-04-26T01:08:56.325 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012630989999999999 2024-04-26T01:08:56.325 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.325 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.325 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.325 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.593958+0000", 2024-04-26T01:08:56.325 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.14e-05 2024-04-26T01:08:56.325 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.326 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.326 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.326 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.326 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.326 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:785 3.5s0 3:a8ed6f44:::smithi16428812-28:head [copy-from ver 62] snapc 15=[15] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:08:56.326 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:38.277776+0000", 2024-04-26T01:08:56.326 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 17.973900676, 2024-04-26T01:08:56.326 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.036482200999999999, 2024-04-26T01:08:56.326 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.326 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.326 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:56.326 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:56.326 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:56.326 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:56.326 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 785 2024-04-26T01:08:56.327 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.327 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.327 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.327 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.327 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277776+0000", 2024-04-26T01:08:56.327 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.327 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.327 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.327 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.327 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277776+0000", 2024-04-26T01:08:56.327 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.327 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.327 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.327 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.328 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277778+0000", 2024-04-26T01:08:56.328 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.866e-06 2024-04-26T01:08:56.328 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.328 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.328 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.328 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277785+0000", 2024-04-26T01:08:56.328 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2629999999999997e-06 2024-04-26T01:08:56.328 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.328 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.328 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.328 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277800+0000", 2024-04-26T01:08:56.328 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4888999999999999e-05 2024-04-26T01:08:56.328 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.328 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.328 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.329 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277807+0000", 2024-04-26T01:08:56.329 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8170000000000001e-06 2024-04-26T01:08:56.329 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.329 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.329 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.329 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277869+0000", 2024-04-26T01:08:56.329 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2844999999999999e-05 2024-04-26T01:08:56.329 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.329 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.329 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:56.329 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277990+0000", 2024-04-26T01:08:56.329 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012097899999999999 2024-04-26T01:08:56.329 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.329 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.329 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:08:56.330 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.302496+0000", 2024-04-26T01:08:56.330 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.024505859000000001 2024-04-26T01:08:56.330 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.330 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.330 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:08:56.330 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.311498+0000", 2024-04-26T01:08:56.330 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0090014369999999993 2024-04-26T01:08:56.330 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.330 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.330 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:08:56.330 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.314125+0000", 2024-04-26T01:08:56.330 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0026272930000000002 2024-04-26T01:08:56.330 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.330 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.331 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.331 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.314258+0000", 2024-04-26T01:08:56.331 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000132953 2024-04-26T01:08:56.331 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.331 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.331 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.331 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.331 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.331 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:786 3.5s0 3:a8ed6f44:::smithi16428812-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:08:56.331 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:38.277827+0000", 2024-04-26T01:08:56.331 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 17.973849253000001, 2024-04-26T01:08:56.331 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.033385495000000001, 2024-04-26T01:08:56.331 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.331 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.331 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:56.332 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:56.332 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:56.332 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:56.332 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 786 2024-04-26T01:08:56.332 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.332 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.332 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.332 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.332 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277827+0000", 2024-04-26T01:08:56.332 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.332 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.332 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.332 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.332 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277827+0000", 2024-04-26T01:08:56.333 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.333 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.333 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.333 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.333 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277828+0000", 2024-04-26T01:08:56.333 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6199999999999997e-07 2024-04-26T01:08:56.333 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.333 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.333 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.333 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277832+0000", 2024-04-26T01:08:56.333 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1400000000000002e-06 2024-04-26T01:08:56.333 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.333 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.333 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.333 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277841+0000", 2024-04-26T01:08:56.334 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0699999999999996e-06 2024-04-26T01:08:56.334 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.334 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.334 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.334 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277848+0000", 2024-04-26T01:08:56.334 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1119999999999998e-06 2024-04-26T01:08:56.334 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.334 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.334 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.334 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.278246+0000", 2024-04-26T01:08:56.334 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00039790799999999999 2024-04-26T01:08:56.334 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.334 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.334 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2024-04-26T01:08:56.335 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.278408+0000", 2024-04-26T01:08:56.335 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016165099999999999 2024-04-26T01:08:56.335 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.335 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.335 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.335 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.310714+0000", 2024-04-26T01:08:56.335 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.032306058999999998 2024-04-26T01:08:56.335 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.335 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.335 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:56.335 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.311023+0000", 2024-04-26T01:08:56.335 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00030885 2024-04-26T01:08:56.335 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.335 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.336 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.336 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.311213+0000", 2024-04-26T01:08:56.336 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00018994300000000001 2024-04-26T01:08:56.336 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.336 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.336 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.336 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.336 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.336 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:863 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1753088,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-04-26T01:08:56.336 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:43.278882+0000", 2024-04-26T01:08:56.336 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 12.972793952, 2024-04-26T01:08:56.336 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.018957168, 2024-04-26T01:08:56.336 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.336 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.337 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:56.337 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:56.337 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:56.337 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:56.337 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 863 2024-04-26T01:08:56.337 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.337 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.337 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.337 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.337 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278882+0000", 2024-04-26T01:08:56.337 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.337 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.337 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.337 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.338 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278882+0000", 2024-04-26T01:08:56.338 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.338 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.338 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.338 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.338 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278886+0000", 2024-04-26T01:08:56.338 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8829999999999999e-06 2024-04-26T01:08:56.338 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.338 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.338 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.338 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278905+0000", 2024-04-26T01:08:56.338 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9168999999999999e-05 2024-04-26T01:08:56.338 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.338 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.339 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.339 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278950+0000", 2024-04-26T01:08:56.339 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5028000000000003e-05 2024-04-26T01:08:56.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.339 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.339 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.339 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278974+0000", 2024-04-26T01:08:56.339 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3682000000000001e-05 2024-04-26T01:08:56.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.339 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.339 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.339 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279142+0000", 2024-04-26T01:08:56.339 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016799499999999999 2024-04-26T01:08:56.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.339 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.340 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2024-04-26T01:08:56.340 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279164+0000", 2024-04-26T01:08:56.340 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2039e-05 2024-04-26T01:08:56.340 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.340 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.340 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.340 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.283284+0000", 2024-04-26T01:08:56.340 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0041193239999999997 2024-04-26T01:08:56.340 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.340 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.340 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:56.340 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.283451+0000", 2024-04-26T01:08:56.340 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016756799999999999 2024-04-26T01:08:56.340 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.340 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.341 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.341 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.297840+0000", 2024-04-26T01:08:56.341 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.01438848 2024-04-26T01:08:56.341 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.341 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.341 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.341 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.341 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.341 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:864 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1753088,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-04-26T01:08:56.341 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:43.279019+0000", 2024-04-26T01:08:56.341 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 12.972657011999999, 2024-04-26T01:08:56.341 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.021489656999999999, 2024-04-26T01:08:56.341 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.341 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.341 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:56.342 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:56.342 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:56.342 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:56.342 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 864 2024-04-26T01:08:56.342 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.342 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.342 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.342 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.342 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279019+0000", 2024-04-26T01:08:56.342 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.342 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.342 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.342 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.342 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279019+0000", 2024-04-26T01:08:56.343 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.343 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.343 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.343 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.343 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279021+0000", 2024-04-26T01:08:56.343 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.243e-06 2024-04-26T01:08:56.343 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.343 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.343 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.343 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279028+0000", 2024-04-26T01:08:56.343 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0899999999999999e-06 2024-04-26T01:08:56.343 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.343 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.343 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.343 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279044+0000", 2024-04-26T01:08:56.344 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6545000000000002e-05 2024-04-26T01:08:56.344 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.344 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.344 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.344 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279053+0000", 2024-04-26T01:08:56.344 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1600000000000004e-06 2024-04-26T01:08:56.344 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.344 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.344 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.344 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279342+0000", 2024-04-26T01:08:56.344 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028902300000000001 2024-04-26T01:08:56.344 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.344 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.344 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:56.344 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279354+0000", 2024-04-26T01:08:56.345 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1542999999999999e-05 2024-04-26T01:08:56.345 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.345 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.345 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.345 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.284845+0000", 2024-04-26T01:08:56.345 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0054915240000000002 2024-04-26T01:08:56.345 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.345 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.345 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:56.345 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.284995+0000", 2024-04-26T01:08:56.345 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014962299999999999 2024-04-26T01:08:56.345 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.345 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.345 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.346 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.300509+0000", 2024-04-26T01:08:56.346 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015513905999999999 2024-04-26T01:08:56.346 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.346 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.346 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.346 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.346 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.346 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:865 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1753088,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-04-26T01:08:56.346 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:43.279067+0000", 2024-04-26T01:08:56.346 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 12.972609255, 2024-04-26T01:08:56.346 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.023511127999999999, 2024-04-26T01:08:56.346 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:08:56.346 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:08:56.346 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:08:56.346 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:08:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:08:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 865 2024-04-26T01:08:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:08:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:08:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279067+0000", 2024-04-26T01:08:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:08:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279067+0000", 2024-04-26T01:08:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:08:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.348 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.348 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:08:56.348 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279068+0000", 2024-04-26T01:08:56.348 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9699999999999995e-07 2024-04-26T01:08:56.348 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.348 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.348 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:08:56.348 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279072+0000", 2024-04-26T01:08:56.348 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2400000000000001e-06 2024-04-26T01:08:56.348 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.348 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.348 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:08:56.348 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279082+0000", 2024-04-26T01:08:56.348 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6430000000000007e-06 2024-04-26T01:08:56.349 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.349 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.349 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:08:56.349 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279089+0000", 2024-04-26T01:08:56.349 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0690000000000004e-06 2024-04-26T01:08:56.349 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.349 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.349 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.349 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279431+0000", 2024-04-26T01:08:56.349 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034206800000000001 2024-04-26T01:08:56.349 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.349 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.349 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:56.349 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279454+0000", 2024-04-26T01:08:56.349 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3448999999999999e-05 2024-04-26T01:08:56.350 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.350 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.350 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:08:56.350 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.287223+0000", 2024-04-26T01:08:56.350 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077688849999999997 2024-04-26T01:08:56.350 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.350 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.350 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:08:56.350 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.287470+0000", 2024-04-26T01:08:56.350 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024656199999999997 2024-04-26T01:08:56.350 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:08:56.350 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:08:56.350 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:08:56.350 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.302578+0000", 2024-04-26T01:08:56.351 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015108415 2024-04-26T01:08:56.351 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.351 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.351 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.351 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:08:56.351 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:08:56.351 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:08:56.352 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:08:56.353 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:56.353+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:56.453 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:56.450+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:56.522 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:56.522 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:56.536 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:08:56.553 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:56.551+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:56.555 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:56.556 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T01:08:56.560 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T01:08:56.654 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:56.653+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:56.700 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:56.700 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:56.714 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:08:56.755 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:56.749+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:56.832 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:56.832 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:56.847 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:08:56.894 INFO:tasks.ceph.osd.0.smithi008.stderr:2024-04-26T01:08:56.892+0000 7fba1fa4f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24997) UID: 0 2024-04-26T01:08:56.957 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:56.957+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:56.983 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:56.983 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:56.997 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:08:57.046 INFO:teuthology.orchestra.run.smithi164.stdout:fsck success 2024-04-26T01:08:57.057 INFO:tasks.thrashosds.thrasher:resharding successfully completed 2024-04-26T01:08:57.058 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-26T01:08:57.058 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-26T01:08:57.058 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-26T01:08:57.058 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-04-26T01:08:57.059 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:57.056+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:57.060 INFO:tasks.rados.rados.0.smithi164.stdout:386: done (1 left) 2024-04-26T01:08:57.060 INFO:tasks.rados.rados.0.smithi164.stdout:388: done (0 left) 2024-04-26T01:08:57.060 INFO:tasks.rados.rados.0.smithi164.stdout:389: rollback oid 36 current snap is 32 2024-04-26T01:08:57.060 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 36 to 24 2024-04-26T01:08:57.060 INFO:tasks.ceph.osd.3:Started 2024-04-26T01:08:57.061 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:08:57.061 INFO:tasks.rados.rados.0.smithi164.stdout:390: read oid 38 snap -1 2024-04-26T01:08:57.061 INFO:tasks.rados.rados.0.smithi164.stdout:390: expect (ObjNum 124 snap 17 seq_num 124) 2024-04-26T01:08:57.061 INFO:tasks.rados.rados.0.smithi164.stdout:391: read oid 24 snap -1 2024-04-26T01:08:57.062 INFO:tasks.rados.rados.0.smithi164.stdout:391: expect (ObjNum 128 snap 19 seq_num 128) 2024-04-26T01:08:57.062 INFO:tasks.rados.rados.0.smithi164.stdout:392: copy_from oid 10 from oid 5 current snap is 32 2024-04-26T01:08:57.062 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 11 current snap is 32 2024-04-26T01:08:57.062 INFO:tasks.rados.rados.0.smithi164.stdout:393: seq_num 155 ranges {2326528=212992} 2024-04-26T01:08:57.080 INFO:tasks.rados.rados.0.smithi164.stdout:393: writing smithi16428812-11 from 2326528 to 2539520 tid 1 2024-04-26T01:08:57.080 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 25 current snap is 32 2024-04-26T01:08:57.080 INFO:tasks.rados.rados.0.smithi164.stdout:389: finishing rollback tid 0 to smithi16428812-36 2024-04-26T01:08:57.080 INFO:tasks.rados.rados.0.smithi164.stdout:389: finishing rollback tid 1 to smithi16428812-36 2024-04-26T01:08:57.080 INFO:tasks.rados.rados.0.smithi164.stdout:389: finishing rollback tid 2 to smithi16428812-36 2024-04-26T01:08:57.080 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 36 v 121 (ObjNum 130 snap 19 seq_num 130) dirty exists 2024-04-26T01:08:57.080 INFO:tasks.rados.rados.0.smithi164.stdout:394: seq_num 156 ranges {2457600=229376} 2024-04-26T01:08:57.101 INFO:tasks.rados.rados.0.smithi164.stdout:394: writing smithi16428812-25 from 2457600 to 2686976 tid 1 2024-04-26T01:08:57.102 INFO:tasks.rados.rados.0.smithi164.stdout:389: done (5 left) 2024-04-26T01:08:57.102 INFO:tasks.rados.rados.0.smithi164.stdout:append oid 35 current snap is 32 2024-04-26T01:08:57.102 INFO:tasks.rados.rados.0.smithi164.stdout:395: seq_num 157 ranges {6094848=245760} 2024-04-26T01:08:57.145 INFO:tasks.rados.rados.0.smithi164.stdout:395: writing smithi16428812-35 from 6094848 to 6340608 tid 1 2024-04-26T01:08:57.145 INFO:tasks.rados.rados.0.smithi164.stdout:396: snap_remove snap 27 2024-04-26T01:08:57.145 INFO:tasks.rados.rados.0.smithi164.stdout:390: expect (ObjNum 124 snap 17 seq_num 124) 2024-04-26T01:08:57.159 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:57.155+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:57.181 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:57.181 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:57.187 INFO:teuthology.orchestra.run.smithi164.stderr:no valid command found; 10 closest matches: 2024-04-26T01:08:57.187 INFO:teuthology.orchestra.run.smithi164.stderr:0 2024-04-26T01:08:57.187 INFO:teuthology.orchestra.run.smithi164.stderr:1 2024-04-26T01:08:57.187 INFO:teuthology.orchestra.run.smithi164.stderr:2 2024-04-26T01:08:57.187 INFO:teuthology.orchestra.run.smithi164.stderr:abort 2024-04-26T01:08:57.187 INFO:teuthology.orchestra.run.smithi164.stderr:assert 2024-04-26T01:08:57.187 INFO:teuthology.orchestra.run.smithi164.stderr:config diff 2024-04-26T01:08:57.187 INFO:teuthology.orchestra.run.smithi164.stderr:config diff get 2024-04-26T01:08:57.187 INFO:teuthology.orchestra.run.smithi164.stderr:config get 2024-04-26T01:08:57.187 INFO:teuthology.orchestra.run.smithi164.stderr:config help [] 2024-04-26T01:08:57.187 INFO:teuthology.orchestra.run.smithi164.stderr:config set ... 2024-04-26T01:08:57.187 INFO:teuthology.orchestra.run.smithi164.stderr:admin_socket: invalid command 2024-04-26T01:08:57.189 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-26T01:08:57.190 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-26T01:08:57.195 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:08:57.200 INFO:tasks.rados.rados.0.smithi164.stdout:393: finishing write tid 1 to smithi16428812-11 2024-04-26T01:08:57.200 INFO:tasks.rados.rados.0.smithi164.stdout:393: finishing write tid 2 to smithi16428812-11 2024-04-26T01:08:57.201 INFO:tasks.rados.rados.0.smithi164.stdout:393: finishing write tid 3 to smithi16428812-11 2024-04-26T01:08:57.201 INFO:tasks.rados.rados.0.smithi164.stdout:393: oid 11 updating version 0 to 39 2024-04-26T01:08:57.201 INFO:tasks.rados.rados.0.smithi164.stdout:393: oid 11 updating version 39 to 40 2024-04-26T01:08:57.201 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 11 v 40 (ObjNum 155 snap 32 seq_num 155) dirty exists 2024-04-26T01:08:57.201 INFO:tasks.rados.rados.0.smithi164.stdout:393: left oid 11 (ObjNum 155 snap 32 seq_num 155) 2024-04-26T01:08:57.201 INFO:tasks.rados.rados.0.smithi164.stdout:392: finishing copy_from racing read to smithi16428812-10 2024-04-26T01:08:57.201 INFO:tasks.rados.rados.0.smithi164.stdout:391: expect (ObjNum 128 snap 19 seq_num 128) 2024-04-26T01:08:57.259 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:57.256+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:57.321 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:57.321 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:57.335 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:08:57.353 INFO:tasks.rados.rados.0.smithi164.stdout:394: finishing write tid 1 to smithi16428812-25 2024-04-26T01:08:57.353 INFO:tasks.rados.rados.0.smithi164.stdout:394: finishing write tid 2 to smithi16428812-25 2024-04-26T01:08:57.353 INFO:tasks.rados.rados.0.smithi164.stdout:394: finishing write tid 3 to smithi16428812-25 2024-04-26T01:08:57.353 INFO:tasks.rados.rados.0.smithi164.stdout:394: oid 25 updating version 0 to 115 2024-04-26T01:08:57.353 INFO:tasks.rados.rados.0.smithi164.stdout:394: oid 25 version 115 is already newer than 114 2024-04-26T01:08:57.354 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 25 v 115 (ObjNum 156 snap 32 seq_num 156) dirty exists 2024-04-26T01:08:57.354 INFO:tasks.rados.rados.0.smithi164.stdout:394: left oid 25 (ObjNum 156 snap 32 seq_num 156) 2024-04-26T01:08:57.354 INFO:tasks.rados.rados.0.smithi164.stdout:395: finishing write tid 1 to smithi16428812-35 2024-04-26T01:08:57.354 INFO:tasks.rados.rados.0.smithi164.stdout:395: finishing write tid 2 to smithi16428812-35 2024-04-26T01:08:57.354 INFO:tasks.rados.rados.0.smithi164.stdout:395: finishing write tid 3 to smithi16428812-35 2024-04-26T01:08:57.354 INFO:tasks.rados.rados.0.smithi164.stdout:395: oid 35 updating version 0 to 87 2024-04-26T01:08:57.354 INFO:tasks.rados.rados.0.smithi164.stdout:395: oid 35 version 87 is already newer than 86 2024-04-26T01:08:57.354 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 35 v 87 (ObjNum 157 snap 32 seq_num 157) dirty exists 2024-04-26T01:08:57.354 INFO:tasks.rados.rados.0.smithi164.stdout:395: left oid 35 (ObjNum 157 snap 32 seq_num 157) 2024-04-26T01:08:57.354 INFO:tasks.rados.rados.0.smithi164.stdout:392: finishing copy_from to smithi16428812-10 2024-04-26T01:08:57.354 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 10 v 32 (ObjNum 127 snap 19 seq_num 127) dirty exists 2024-04-26T01:08:57.378 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:57.376+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:57.462 INFO:tasks.ceph.osd.15.smithi164.stderr:2024-04-26T01:08:57.458+0000 7f0ae0d36640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25270) UID: 0 2024-04-26T01:08:57.506 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:57.506 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:57.520 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:08:57.563 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:57.563+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:57.647 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:57.647 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:57.661 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:08:57.663 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:57.662+0000 7ff2871de640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 32810) UID: 0 2024-04-26T01:08:57.688 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:08:57.686+0000 7fd70ac967c0 -1 Falling back to public interface 2024-04-26T01:08:57.765 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:57.762+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:57.788 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:57.789 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:57.803 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:08:57.865 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:57.865+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:57.922 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:57.922 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:57.937 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:08:57.966 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-26T01:08:57.963+0000 7fe1b275f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25052) UID: 0 2024-04-26T01:08:58.063 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:58.063 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:58.065 INFO:tasks.rados.rados.0.smithi164.stdout:390: done (6 left) 2024-04-26T01:08:58.065 INFO:tasks.rados.rados.0.smithi164.stdout:391: done (5 left) 2024-04-26T01:08:58.065 INFO:tasks.rados.rados.0.smithi164.stdout:392: done (4 left) 2024-04-26T01:08:58.065 INFO:tasks.rados.rados.0.smithi164.stdout:393: done (3 left) 2024-04-26T01:08:58.065 INFO:tasks.rados.rados.0.smithi164.stdout:394: done (2 left) 2024-04-26T01:08:58.065 INFO:tasks.rados.rados.0.smithi164.stdout:395: done (1 left) 2024-04-26T01:08:58.065 INFO:tasks.rados.rados.0.smithi164.stdout:396: done (0 left) 2024-04-26T01:08:58.065 INFO:tasks.rados.rados.0.smithi164.stdout:397: read oid 23 snap -1 2024-04-26T01:08:58.065 INFO:tasks.rados.rados.0.smithi164.stdout:397: expect deleted 2024-04-26T01:08:58.065 INFO:tasks.rados.rados.0.smithi164.stdout:398: rollback oid 20 current snap is 32 2024-04-26T01:08:58.065 INFO:tasks.rados.rados.0.smithi164.stdout:rollback oid 20 to 30 2024-04-26T01:08:58.065 INFO:tasks.rados.rados.0.smithi164.stdout:399: copy_from oid 46 from oid 10 current snap is 32 2024-04-26T01:08:58.066 INFO:tasks.rados.rados.0.smithi164.stdout: waiting on 3 2024-04-26T01:08:58.067 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:58.066+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:58.071 INFO:tasks.rados.rados.0.smithi164.stdout:397: done (2 left) 2024-04-26T01:08:58.071 INFO:tasks.rados.rados.0.smithi164.stdout: waiting on 2 2024-04-26T01:08:58.085 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:08:58.086 INFO:tasks.rados.rados.0.smithi164.stdout:398: finishing rollback tid 0 to smithi16428812-20 2024-04-26T01:08:58.086 INFO:tasks.rados.rados.0.smithi164.stdout:398: finishing rollback tid 1 to smithi16428812-20 2024-04-26T01:08:58.087 INFO:tasks.rados.rados.0.smithi164.stdout:398: finishing rollback tid 2 to smithi16428812-20 2024-04-26T01:08:58.087 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 20 v 20 (ObjNum 69 snap 4 seq_num 69) dirty exists 2024-04-26T01:08:58.087 INFO:tasks.rados.rados.0.smithi164.stdout:398: done (1 left) 2024-04-26T01:08:58.087 INFO:tasks.rados.rados.0.smithi164.stdout: waiting on 1 2024-04-26T01:08:58.108 INFO:tasks.rados.rados.0.smithi164.stdout:399: finishing copy_from racing read to smithi16428812-46 2024-04-26T01:08:58.109 INFO:tasks.rados.rados.0.smithi164.stdout:399: finishing copy_from to smithi16428812-46 2024-04-26T01:08:58.109 INFO:tasks.rados.rados.0.smithi164.stdout:update_object_version oid 46 v 32 (ObjNum 127 snap 19 seq_num 127) dirty exists 2024-04-26T01:08:58.109 INFO:tasks.rados.rados.0.smithi164.stdout:399: done (0 left) 2024-04-26T01:08:58.118 INFO:tasks.rados.rados.0.smithi164.stderr:0 errors. 2024-04-26T01:08:58.118 INFO:tasks.rados.rados.0.smithi164.stderr: 2024-04-26T01:08:58.125 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T01:08:58.168 INFO:tasks.ceph.osd.10.smithi116.stderr:2024-04-26T01:08:58.167+0000 7f67bbba3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25058) UID: 0 2024-04-26T01:08:58.207 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:58.207 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:58.223 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:08:58.269 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:58.268+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:58.351 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:58.351 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:58.366 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:08:58.410 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:58.408+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:58.433 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:08:58.433 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":92,"fsid":"4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554","created":"2024-04-26T01:07:31.350368+0000","modified":"2024-04-26T01:08:58.057451+0000","last_up_change":"2024-04-26T01:08:46.295028+0000","last_in_change":"2024-04-26T01:08:47.301350+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:07:44.085247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T01:07:46.423491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-04-26T01:07:55.307661+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":92,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"abed240a-e896-459b-8a43-cd1419cafbc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6801","nonce":188308042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6803","nonce":188308042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6807","nonce":188308042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6805","nonce":188308042}]},"public_addr":"172.21.15.8:6801/188308042","cluster_addr":"172.21.15.8:6803/188308042","heartbeat_back_addr":"172.21.15.8:6807/188308042","heartbeat_front_addr":"172.21.15.8:6805/188308042","state":["exists","up"]},{"osd":1,"uuid":"831c717c-e124-4489-a44f-93db78c47deb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6819","nonce":3487767810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6823","nonce":3487767810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6831","nonce":3487767810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6828","nonce":3487767810}]},"public_addr":"172.21.15.12:6819/3487767810","cluster_addr":"172.21.15.12:6823/3487767810","heartbeat_back_addr":"172.21.15.12:6831/3487767810","heartbeat_front_addr":"172.21.15.12:6828/3487767810","state":["exists","up"]},{"osd":2,"uuid":"92a95165-ae34-4f29-a7f3-fe97afc26af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6803","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6805","nonce":3599623558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6807","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6809","nonce":3599623558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6815","nonce":3599623558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6813","nonce":3599623558}]},"public_addr":"172.21.15.116:6805/3599623558","cluster_addr":"172.21.15.116:6809/3599623558","heartbeat_back_addr":"172.21.15.116:6815/3599623558","heartbeat_front_addr":"172.21.15.116:6813/3599623558","state":["exists","up"]},{"osd":3,"uuid":"e9d8f764-4951-4de4-92c7-682850852e85","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":80,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6804","nonce":4123918458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6808","nonce":4123918458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6816","nonce":4123918458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":4123918458},{"type":"v1","addr":"172.21.15.164:6812","nonce":4123918458}]},"public_addr":"172.21.15.164:6804/4123918458","cluster_addr":"172.21.15.164:6808/4123918458","heartbeat_back_addr":"172.21.15.164:6816/4123918458","heartbeat_front_addr":"172.21.15.164:6812/4123918458","state":["exists"]},{"osd":4,"uuid":"7f5e93f8-b1f6-41e7-8da9-b8e57d4259bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6825","nonce":425515720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6827","nonce":425515720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6831","nonce":425515720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6829","nonce":425515720}]},"public_addr":"172.21.15.8:6825/425515720","cluster_addr":"172.21.15.8:6827/425515720","heartbeat_back_addr":"172.21.15.8:6831/425515720","heartbeat_front_addr":"172.21.15.8:6829/425515720","state":["exists","up"]},{"osd":5,"uuid":"f7b3a311-3bf6-4ae5-b7d2-dedbbe6c47ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6809","nonce":1641884791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6811","nonce":1641884791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6815","nonce":1641884791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6813","nonce":1641884791}]},"public_addr":"172.21.15.12:6809/1641884791","cluster_addr":"172.21.15.12:6811/1641884791","heartbeat_back_addr":"172.21.15.12:6815/1641884791","heartbeat_front_addr":"172.21.15.12:6813/1641884791","state":["exists","up"]},{"osd":6,"uuid":"796d7212-13de-43b2-8aa4-4abc4fd210a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6817","nonce":156763037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6820","nonce":156763037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6825","nonce":156763037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6821","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6823","nonce":156763037}]},"public_addr":"172.21.15.116:6817/156763037","cluster_addr":"172.21.15.116:6820/156763037","heartbeat_back_addr":"172.21.15.116:6825/156763037","heartbeat_front_addr":"172.21.15.116:6823/156763037","state":["exists","up"]},{"osd":7,"uuid":"ccb8cd0f-a0d6-44d7-86cc-809eff9687ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6826","nonce":2291946233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6829","nonce":2291946233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6833","nonce":2291946233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6831","nonce":2291946233}]},"public_addr":"172.21.15.164:6826/2291946233","cluster_addr":"172.21.15.164:6829/2291946233","heartbeat_back_addr":"172.21.15.164:6833/2291946233","heartbeat_front_addr":"172.21.15.164:6831/2291946233","state":["exists","up"]},{"osd":8,"uuid":"bf13e311-71b7-44ff-af1b-68d118935c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6809","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6811","nonce":2709504491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6813","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6815","nonce":2709504491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6821","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6823","nonce":2709504491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6817","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6819","nonce":2709504491}]},"public_addr":"172.21.15.8:6811/2709504491","cluster_addr":"172.21.15.8:6815/2709504491","heartbeat_back_addr":"172.21.15.8:6823/2709504491","heartbeat_front_addr":"172.21.15.8:6819/2709504491","state":["exists","up"]},{"osd":9,"uuid":"4864f46f-fd7b-48fc-908b-2564f7bda67b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6801","nonce":1155130892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6803","nonce":1155130892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6807","nonce":1155130892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6805","nonce":1155130892}]},"public_addr":"172.21.15.12:6801/1155130892","cluster_addr":"172.21.15.12:6803/1155130892","heartbeat_back_addr":"172.21.15.12:6807/1155130892","heartbeat_front_addr":"172.21.15.12:6805/1155130892","state":["exists","up"]},{"osd":10,"uuid":"09cbb9ca-f609-409e-821b-93e695b87067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6822","nonce":1461831728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6827","nonce":1461831728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6831","nonce":1461831728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6829","nonce":1461831728}]},"public_addr":"172.21.15.116:6822/1461831728","cluster_addr":"172.21.15.116:6827/1461831728","heartbeat_back_addr":"172.21.15.116:6831/1461831728","heartbeat_front_addr":"172.21.15.116:6829/1461831728","state":["exists","up"]},{"osd":11,"uuid":"9bfdf353-5174-40dd-bb3c-ab2dc850adad","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":42,"up_from":60,"up_thru":30,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6819","nonce":838948348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6821","nonce":838948348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6825","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6827","nonce":838948348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6823","nonce":838948348}]},"public_addr":"172.21.15.164:6819/838948348","cluster_addr":"172.21.15.164:6821/838948348","heartbeat_back_addr":"172.21.15.164:6827/838948348","heartbeat_front_addr":"172.21.15.164:6823/838948348","state":["exists","up"]},{"osd":12,"uuid":"a3e244d8-a765-467e-820c-caa5d1d97c9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6810","nonce":2082773890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6814","nonce":2082773890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6822","nonce":2082773890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6818","nonce":2082773890}]},"public_addr":"172.21.15.8:6810/2082773890","cluster_addr":"172.21.15.8:6814/2082773890","heartbeat_back_addr":"172.21.15.8:6822/2082773890","heartbeat_front_addr":"172.21.15.8:6818/2082773890","state":["exists","up"]},{"osd":13,"uuid":"383875c2-0a0c-4834-8beb-a6784eef1f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6817","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6818","nonce":3158583580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6822","nonce":3158583580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6829","nonce":3158583580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6826","nonce":3158583580}]},"public_addr":"172.21.15.12:6818/3158583580","cluster_addr":"172.21.15.12:6822/3158583580","heartbeat_back_addr":"172.21.15.12:6829/3158583580","heartbeat_front_addr":"172.21.15.12:6826/3158583580","state":["exists","up"]},{"osd":14,"uuid":"30311d57-c625-4dff-98d5-1ba0a84c0ceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6801","nonce":2158650188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6804","nonce":2158650188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6812","nonce":2158650188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6808","nonce":2158650188}]},"public_addr":"172.21.15.116:6801/2158650188","cluster_addr":"172.21.15.116:6804/2158650188","heartbeat_back_addr":"172.21.15.116:6812/2158650188","heartbeat_front_addr":"172.21.15.116:6808/2158650188","state":["exists","up"]},{"osd":15,"uuid":"1243478f-9908-460d-88d8-3c542d5258b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6805","nonce":2601606699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6809","nonce":2601606699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6817","nonce":2601606699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6813","nonce":2601606699}]},"public_addr":"172.21.15.164:6805/2601606699","cluster_addr":"172.21.15.164:6809/2601606699","heartbeat_back_addr":"172.21.15.164:6817/2601606699","heartbeat_front_addr":"172.21.15.164:6813/2601606699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:43.889257+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:46.000388+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.437264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-26T01:08:43.947103+0000","dead_epoch":80},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.474428+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:44.951276+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:47.952421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.501842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:51.155841+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.492788+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.473510+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-26T01:07:41.411335+0000","dead_epoch":43},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.316127+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.506103+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:47.436227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:07:41.338969+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":21,"length":1},{"begin":24,"length":2},{"begin":28,"length":3}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":29,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:08:58.454 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:08:58.470 INFO:tasks.ceph.osd.9.smithi012.stderr:2024-04-26T01:08:58.465+0000 7efd3daec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24944) UID: 0 2024-04-26T01:08:58.532 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:58.532 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:58.547 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:08:58.561 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:08:58.570 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-26T01:08:58.565+0000 7f3bc37ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24940) UID: 0 2024-04-26T01:08:58.672 INFO:tasks.ceph.osd.11.smithi164.stderr:2024-04-26T01:08:58.670+0000 7ff2871de640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 32810) UID: 0 2024-04-26T01:08:58.744 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-04-26T01:08:58.745 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:08:58.753 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-04-26T01:08:58.753 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:08:58.769 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-04-26T01:08:58.770 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:08:58.772 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:08:58.773 INFO:tasks.ceph.osd.13.smithi012.stderr:2024-04-26T01:08:58.769+0000 7f5c94210640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24946) UID: 0 2024-04-26T01:08:58.795 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-04-26T01:08:58.795 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:08:58.795 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-26T01:08:58.816 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-04-26T01:08:58.817 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:08:58.840 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-04-26T01:08:58.840 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:08:58.865 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-04-26T01:08:58.865 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:08:58.874 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-04-26T01:08:58.871+0000 7f0f6d910640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25056) UID: 0 2024-04-26T01:08:58.876 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:08:58.877 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-26T01:08:58.891 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-04-26T01:08:58.892 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:08:58.895 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-04-26T01:08:58.063641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":105251360,"num_objects":60,"num_object_clones":15,"num_object_copies":410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":2,"num_read":602,"num_read_kb":204856,"num_write":1032,"num_write_kb":169796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":33882112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1188,"ondisk_log_size":1188,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":14,"num_per_pool_osds":14,"num_per_pool_omap_osds":14,"kb":1321205760,"kb_used":639080,"kb_used_data":192776,"kb_used_omap":21,"kb_used_meta":446058,"kb_avail":1320566680,"statfs":{"total":1352914698240,"available":1352260280320,"internally_reserved":0,"allocated":197402624,"data_stored":244530289,"data_compressed":2785136,"data_compressed_allocated":3821568,"data_compressed_original":8503296,"omap_allocated":22333,"internal_metadata":456763587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":97,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009320"},"pg_stats":[{"pgid":"3.f","version":"86'117","reported_seq":339,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.126412+0000","last_change":"2024-04-26T01:08:52.115080+0000","last_active":"2024-04-26T01:08:53.126412+0000","last_peered":"2024-04-26T01:08:53.126412+0000","last_clean":"2024-04-26T01:08:53.126412+0000","last_became_active":"2024-04-26T01:08:11.056866+0000","last_became_peered":"2024-04-26T01:08:11.056866+0000","last_unstale":"2024-04-26T01:08:53.126412+0000","last_undegraded":"2024-04-26T01:08:53.126412+0000","last_fullsized":"2024-04-26T01:08:53.126412+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.632185+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.028987595000000001,"stat_sum":{"num_bytes":16072704,"num_objects":6,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":17589,"num_write":89,"num_write_kb":19072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3129344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,14,13,8,10,0],"acting":[4,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"3.e","version":"85'64","reported_seq":376,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.126680+0000","last_change":"2024-04-26T01:08:52.096444+0000","last_active":"2024-04-26T01:08:53.126680+0000","last_peered":"2024-04-26T01:08:53.126680+0000","last_clean":"2024-04-26T01:08:53.126680+0000","last_became_active":"2024-04-26T01:08:11.054010+0000","last_became_peered":"2024-04-26T01:08:11.054010+0000","last_unstale":"2024-04-26T01:08:53.126680+0000","last_undegraded":"2024-04-26T01:08:53.126680+0000","last_fullsized":"2024-04-26T01:08:53.126680+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:14.640249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015273280000000001,"stat_sum":{"num_bytes":7340032,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":16042,"num_write":28,"num_write_kb":9856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3112960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,7,8,6,12],"acting":[2,9,1,7,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"3.d","version":"82'31","reported_seq":360,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T01:08:54.058055+0000","last_change":"2024-04-26T01:08:54.058827+0000","last_active":"2024-04-26T01:08:54.058055+0000","last_peered":"2024-04-26T01:08:54.058055+0000","last_clean":"2024-04-26T01:08:54.058055+0000","last_became_active":"2024-04-26T01:08:48.340982+0000","last_became_peered":"2024-04-26T01:08:48.340982+0000","last_unstale":"2024-04-26T01:08:54.058055+0000","last_undegraded":"2024-04-26T01:08:54.058055+0000","last_fullsized":"2024-04-26T01:08:54.058055+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:57.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087995,"stat_sum":{"num_bytes":1490944,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":872,"num_write":15,"num_write_kb":1648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1523712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,14,10,9,8,4],"acting":[7,0,14,10,9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"3.c","version":"86'104","reported_seq":331,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.129854+0000","last_change":"2024-04-26T01:08:52.113674+0000","last_active":"2024-04-26T01:08:53.129854+0000","last_peered":"2024-04-26T01:08:53.129854+0000","last_clean":"2024-04-26T01:08:53.129854+0000","last_became_active":"2024-04-26T01:08:11.076105+0000","last_became_peered":"2024-04-26T01:08:11.076105+0000","last_unstale":"2024-04-26T01:08:53.129854+0000","last_undegraded":"2024-04-26T01:08:53.129854+0000","last_fullsized":"2024-04-26T01:08:53.129854+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:14.806843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021254756999999999,"stat_sum":{"num_bytes":11059200,"num_objects":6,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":33525,"num_write":88,"num_write_kb":11888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4849664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0,10,6,12,5],"acting":[9,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"3.b","version":"86'87","reported_seq":378,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.125937+0000","last_change":"2024-04-26T01:08:52.103155+0000","last_active":"2024-04-26T01:08:53.125937+0000","last_peered":"2024-04-26T01:08:53.125937+0000","last_clean":"2024-04-26T01:08:53.125937+0000","last_became_active":"2024-04-26T01:07:57.596133+0000","last_became_peered":"2024-04-26T01:07:57.596133+0000","last_unstale":"2024-04-26T01:08:53.125937+0000","last_undegraded":"2024-04-26T01:08:53.125937+0000","last_fullsized":"2024-04-26T01:08:53.125937+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:59.370444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016820634000000001,"stat_sum":{"num_bytes":9650176,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":18874,"num_write":56,"num_write_kb":12848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"3.a","version":"82'16","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.119769+0000","last_change":"2024-04-26T01:08:52.107071+0000","last_active":"2024-04-26T01:08:53.119769+0000","last_peered":"2024-04-26T01:08:53.119769+0000","last_clean":"2024-04-26T01:08:53.119769+0000","last_became_active":"2024-04-26T01:08:48.328385+0000","last_became_peered":"2024-04-26T01:08:48.328385+0000","last_unstale":"2024-04-26T01:08:53.119769+0000","last_undegraded":"2024-04-26T01:08:53.119769+0000","last_fullsized":"2024-04-26T01:08:53.119769+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:07.168241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001521957,"stat_sum":{"num_bytes":1490944,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":8740,"num_write":7,"num_write_kb":1456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4472832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,1,9,14,6,10],"acting":[8,7,1,9,14,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"3.9","version":"0'0","reported_seq":286,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.128566+0000","last_change":"2024-04-26T01:08:52.100798+0000","last_active":"2024-04-26T01:08:53.128566+0000","last_peered":"2024-04-26T01:08:53.128566+0000","last_clean":"2024-04-26T01:08:53.128566+0000","last_became_active":"2024-04-26T01:08:48.329084+0000","last_became_peered":"2024-04-26T01:08:48.329084+0000","last_unstale":"2024-04-26T01:08:53.128566+0000","last_undegraded":"2024-04-26T01:08:53.128566+0000","last_fullsized":"2024-04-26T01:08:53.128566+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:20.220824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001394476,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,12,15,8,10],"acting":[14,1,0,12,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"3.8","version":"85'56","reported_seq":361,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T01:08:54.057392+0000","last_change":"2024-04-26T01:08:54.058293+0000","last_active":"2024-04-26T01:08:54.057392+0000","last_peered":"2024-04-26T01:08:54.057392+0000","last_clean":"2024-04-26T01:08:54.057392+0000","last_became_active":"2024-04-26T01:07:57.586870+0000","last_became_peered":"2024-04-26T01:07:57.586870+0000","last_unstale":"2024-04-26T01:08:54.057392+0000","last_undegraded":"2024-04-26T01:08:54.057392+0000","last_fullsized":"2024-04-26T01:08:54.057392+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:18.751422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001384751,"stat_sum":{"num_bytes":4603904,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":9202,"num_write":28,"num_write_kb":10400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"1.0","version":"27'157","reported_seq":333,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T01:08:54.083994+0000","last_change":"2024-04-26T01:07:46.409257+0000","last_active":"2024-04-26T01:08:54.083994+0000","last_peered":"2024-04-26T01:08:54.083994+0000","last_clean":"2024-04-26T01:08:54.083994+0000","last_became_active":"2024-04-26T01:07:46.407301+0000","last_became_peered":"2024-04-26T01:07:46.407301+0000","last_unstale":"2024-04-26T01:08:54.083994+0000","last_undegraded":"2024-04-26T01:08:54.083994+0000","last_fullsized":"2024-04-26T01:08:54.083994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_clean_scrub_stamp":"2024-04-26T01:07:45.031420+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.371437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":134,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T01:08:54.056737+0000","last_change":"2024-04-26T01:07:48.145129+0000","last_active":"2024-04-26T01:08:54.056737+0000","last_peered":"2024-04-26T01:08:54.056737+0000","last_clean":"2024-04-26T01:08:54.056737+0000","last_became_active":"2024-04-26T01:07:48.144313+0000","last_became_peered":"2024-04-26T01:07:48.144313+0000","last_unstale":"2024-04-26T01:08:54.056737+0000","last_undegraded":"2024-04-26T01:08:54.056737+0000","last_fullsized":"2024-04-26T01:08:54.056737+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:03.023694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"73'37","reported_seq":377,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T01:08:54.082954+0000","last_change":"2024-04-26T01:08:54.085380+0000","last_active":"2024-04-26T01:08:54.082954+0000","last_peered":"2024-04-26T01:08:54.082954+0000","last_clean":"2024-04-26T01:08:54.082954+0000","last_became_active":"2024-04-26T01:08:48.328645+0000","last_became_peered":"2024-04-26T01:08:48.328645+0000","last_unstale":"2024-04-26T01:08:54.082954+0000","last_undegraded":"2024-04-26T01:08:54.082954+0000","last_fullsized":"2024-04-26T01:08:54.082954+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:21.443771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030000539999999998,"stat_sum":{"num_bytes":4866048,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":1832,"num_write":44,"num_write_kb":6208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5259264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,7,6,15,1,14],"acting":[13,2,7,6,15,1,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.087766+0000","last_change":"2024-04-26T01:08:48.331876+0000","last_active":"2024-04-26T01:08:53.087766+0000","last_peered":"2024-04-26T01:08:53.087766+0000","last_clean":"2024-04-26T01:08:53.087766+0000","last_became_active":"2024-04-26T01:08:48.328976+0000","last_became_peered":"2024-04-26T01:08:48.328976+0000","last_unstale":"2024-04-26T01:08:53.087766+0000","last_undegraded":"2024-04-26T01:08:53.087766+0000","last_fullsized":"2024-04-26T01:08:53.087766+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:07.668608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"86'112","reported_seq":432,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.125569+0000","last_change":"2024-04-26T01:08:52.102769+0000","last_active":"2024-04-26T01:08:53.125569+0000","last_peered":"2024-04-26T01:08:53.125569+0000","last_clean":"2024-04-26T01:08:53.125569+0000","last_became_active":"2024-04-26T01:08:11.061821+0000","last_became_peered":"2024-04-26T01:08:11.061821+0000","last_unstale":"2024-04-26T01:08:53.125569+0000","last_undegraded":"2024-04-26T01:08:53.125569+0000","last_fullsized":"2024-04-26T01:08:53.125569+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:05.783511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017154856,"stat_sum":{"num_bytes":7274496,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":36,"num_read_kb":8848,"num_write":115,"num_write_kb":14848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3522560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,9,5,10,2,13],"acting":[4,7,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":133,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T01:08:54.085004+0000","last_change":"2024-04-26T01:07:48.146870+0000","last_active":"2024-04-26T01:08:54.085004+0000","last_peered":"2024-04-26T01:08:54.085004+0000","last_clean":"2024-04-26T01:08:54.085004+0000","last_became_active":"2024-04-26T01:07:48.145991+0000","last_became_peered":"2024-04-26T01:07:48.145991+0000","last_unstale":"2024-04-26T01:08:54.085004+0000","last_undegraded":"2024-04-26T01:08:54.085004+0000","last_fullsized":"2024-04-26T01:08:54.085004+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:13.523679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"81'80","reported_seq":376,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.128604+0000","last_change":"2024-04-26T01:08:52.102715+0000","last_active":"2024-04-26T01:08:53.128604+0000","last_peered":"2024-04-26T01:08:53.128604+0000","last_clean":"2024-04-26T01:08:53.128604+0000","last_became_active":"2024-04-26T01:07:57.587743+0000","last_became_peered":"2024-04-26T01:07:57.587743+0000","last_unstale":"2024-04-26T01:08:53.128604+0000","last_undegraded":"2024-04-26T01:08:53.128604+0000","last_fullsized":"2024-04-26T01:08:53.128604+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:10.429282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001638207,"stat_sum":{"num_bytes":11862016,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":18810,"num_write":82,"num_write_kb":20144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.051200+0000","last_change":"2024-04-26T01:08:48.324745+0000","last_active":"2024-04-26T01:08:53.051200+0000","last_peered":"2024-04-26T01:08:53.051200+0000","last_clean":"2024-04-26T01:08:53.051200+0000","last_became_active":"2024-04-26T01:08:48.324227+0000","last_became_peered":"2024-04-26T01:08:48.324227+0000","last_unstale":"2024-04-26T01:08:53.051200+0000","last_undegraded":"2024-04-26T01:08:53.051200+0000","last_fullsized":"2024-04-26T01:08:53.051200+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.270010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,5],"acting":[12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.3","version":"85'28","reported_seq":336,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T01:08:54.085416+0000","last_change":"2024-04-26T01:08:54.086168+0000","last_active":"2024-04-26T01:08:54.085416+0000","last_peered":"2024-04-26T01:08:54.085416+0000","last_clean":"2024-04-26T01:08:54.085416+0000","last_became_active":"2024-04-26T01:08:48.332398+0000","last_became_peered":"2024-04-26T01:08:48.332398+0000","last_unstale":"2024-04-26T01:08:54.085416+0000","last_undegraded":"2024-04-26T01:08:54.085416+0000","last_fullsized":"2024-04-26T01:08:54.085416+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:03.235251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001763826,"stat_sum":{"num_bytes":3817472,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":4325,"num_write":36,"num_write_kb":5264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7012352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,6],"acting":[13,7,4,15,2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"3.5","version":"85'60","reported_seq":345,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.588697+0000","last_change":"2024-04-26T01:08:52.059160+0000","last_active":"2024-04-26T01:08:53.588697+0000","last_peered":"2024-04-26T01:08:53.588697+0000","last_clean":"2024-04-26T01:08:53.588697+0000","last_became_active":"2024-04-26T01:07:57.588984+0000","last_became_peered":"2024-04-26T01:07:57.588984+0000","last_unstale":"2024-04-26T01:08:53.588697+0000","last_undegraded":"2024-04-26T01:08:53.588697+0000","last_fullsized":"2024-04-26T01:08:53.588697+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:07.985123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011745200000000001,"stat_sum":{"num_bytes":1753088,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":20,"num_read_kb":17724,"num_write":26,"num_write_kb":10784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":139,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.126222+0000","last_change":"2024-04-26T01:08:50.478667+0000","last_active":"2024-04-26T01:08:53.126222+0000","last_peered":"2024-04-26T01:08:53.126222+0000","last_clean":"2024-04-26T01:08:53.126222+0000","last_became_active":"2024-04-26T01:07:48.465232+0000","last_became_peered":"2024-04-26T01:07:48.465232+0000","last_unstale":"2024-04-26T01:08:53.126222+0000","last_undegraded":"2024-04-26T01:08:53.126222+0000","last_fullsized":"2024-04-26T01:08:53.126222+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:08:50.478571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:08:50.478571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:08.707444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"87'69","reported_seq":361,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.130236+0000","last_change":"2024-04-26T01:08:52.109481+0000","last_active":"2024-04-26T01:08:53.130236+0000","last_peered":"2024-04-26T01:08:53.130236+0000","last_clean":"2024-04-26T01:08:53.130236+0000","last_became_active":"2024-04-26T01:07:57.578181+0000","last_became_peered":"2024-04-26T01:07:57.578181+0000","last_unstale":"2024-04-26T01:08:53.130236+0000","last_undegraded":"2024-04-26T01:08:53.130236+0000","last_fullsized":"2024-04-26T01:08:53.130236+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:17.786505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026039190000000001,"stat_sum":{"num_bytes":6537216,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":7189,"num_write":72,"num_write_kb":6896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":132,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.087857+0000","last_change":"2024-04-26T01:07:48.145645+0000","last_active":"2024-04-26T01:08:53.087857+0000","last_peered":"2024-04-26T01:08:53.087857+0000","last_clean":"2024-04-26T01:08:53.087857+0000","last_became_active":"2024-04-26T01:07:48.145376+0000","last_became_peered":"2024-04-26T01:07:48.145376+0000","last_unstale":"2024-04-26T01:08:53.087857+0000","last_undegraded":"2024-04-26T01:08:53.087857+0000","last_fullsized":"2024-04-26T01:08:53.087857+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:12.310599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"86'97","reported_seq":445,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.126569+0000","last_change":"2024-04-26T01:08:52.104196+0000","last_active":"2024-04-26T01:08:53.126569+0000","last_peered":"2024-04-26T01:08:53.126569+0000","last_clean":"2024-04-26T01:08:53.126569+0000","last_became_active":"2024-04-26T01:08:11.064770+0000","last_became_peered":"2024-04-26T01:08:11.064770+0000","last_unstale":"2024-04-26T01:08:53.126569+0000","last_undegraded":"2024-04-26T01:08:53.126569+0000","last_fullsized":"2024-04-26T01:08:53.126569+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:24.397031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018131675,"stat_sum":{"num_bytes":5308416,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":84,"num_read_kb":31876,"num_write":82,"num_write_kb":21920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,14],"acting":[4,6,9,0,8,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":131,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.119958+0000","last_change":"2024-04-26T01:07:48.145909+0000","last_active":"2024-04-26T01:08:53.119958+0000","last_peered":"2024-04-26T01:08:53.119958+0000","last_clean":"2024-04-26T01:08:53.119958+0000","last_became_active":"2024-04-26T01:07:48.145512+0000","last_became_peered":"2024-04-26T01:07:48.145512+0000","last_unstale":"2024-04-26T01:08:53.119958+0000","last_undegraded":"2024-04-26T01:08:53.119958+0000","last_fullsized":"2024-04-26T01:08:53.119958+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:16.344494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"85'73","reported_seq":368,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.124450+0000","last_change":"2024-04-26T01:08:52.098295+0000","last_active":"2024-04-26T01:08:53.124450+0000","last_peered":"2024-04-26T01:08:53.124450+0000","last_clean":"2024-04-26T01:08:53.124450+0000","last_became_active":"2024-04-26T01:07:57.597503+0000","last_became_peered":"2024-04-26T01:07:57.597503+0000","last_unstale":"2024-04-26T01:08:53.124450+0000","last_undegraded":"2024-04-26T01:08:53.124450+0000","last_fullsized":"2024-04-26T01:08:53.124450+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:56.266998+0000","last_clean_scrub_stamp":"2024-04-26T01:07:56.266998+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:58.113700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001990407,"stat_sum":{"num_bytes":11534336,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":9290,"num_write":61,"num_write_kb":13104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":130,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:08:53.588416+0000","last_change":"2024-04-26T01:07:48.149017+0000","last_active":"2024-04-26T01:08:53.588416+0000","last_peered":"2024-04-26T01:08:53.588416+0000","last_clean":"2024-04-26T01:08:53.588416+0000","last_became_active":"2024-04-26T01:07:48.148023+0000","last_became_peered":"2024-04-26T01:07:48.148023+0000","last_unstale":"2024-04-26T01:08:53.588416+0000","last_undegraded":"2024-04-26T01:08:53.588416+0000","last_fullsized":"2024-04-26T01:08:53.588416+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:07:47.115876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:07.554062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":104660992,"num_objects":58,"num_object_clones":15,"num_object_copies":406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":2,"num_read":466,"num_read_kb":204738,"num_write":829,"num_write_kb":166336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":33882112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":190464000,"data_stored":241639424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1802,"internal_metadata":0},"log_size":1031,"ondisk_log_size":1031,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":23216,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":21,"seq":90194313234,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40028,"kb_used_data":13004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331812,"statfs":{"total":96636764160,"available":96595775488,"internally_reserved":0,"allocated":13316096,"data_stored":17123994,"data_compressed":199708,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,4,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280528,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44896,"kb_used_data":13708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326944,"statfs":{"total":96636764160,"available":96590790656,"internally_reserved":0,"allocated":14036992,"data_stored":17358010,"data_compressed":211316,"data_compressed_allocated":311296,"data_compressed_original":1052672,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280530,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42992,"kb_used_data":11804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328848,"statfs":{"total":96636764160,"available":96592740352,"internally_reserved":0,"allocated":12087296,"data_stored":15567514,"data_compressed":199708,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,8,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313232,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70972,"kb_used_data":17580,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94300868,"statfs":{"total":96636764160,"available":96564088832,"internally_reserved":0,"allocated":18001920,"data_stored":23700017,"data_compressed":196617,"data_compressed_allocated":266240,"data_compressed_original":532480,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313233,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47448,"kb_used_data":16264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324392,"statfs":{"total":96636764160,"available":96588177408,"internally_reserved":0,"allocated":16654336,"data_stored":19244113,"data_compressed":208225,"data_compressed_allocated":307200,"data_compressed_original":1044480,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,4,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313234,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41544,"kb_used_data":10360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330296,"statfs":{"total":96636764160,"available":96594223104,"internally_reserved":0,"allocated":10608640,"data_stored":12190257,"data_compressed":196617,"data_compressed_allocated":266240,"data_compressed_original":532480,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313232,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47724,"kb_used_data":16544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324116,"statfs":{"total":96636764160,"available":96587894784,"internally_reserved":0,"allocated":16941056,"data_stored":20554289,"data_compressed":196617,"data_compressed_allocated":266240,"data_compressed_original":532480,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,6,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313232,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47236,"kb_used_data":16052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324604,"statfs":{"total":96636764160,"available":96588394496,"internally_reserved":0,"allocated":16437248,"data_stored":19804721,"data_compressed":196617,"data_compressed_allocated":266240,"data_compressed_original":532480,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":21,"seq":90194313232,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44620,"kb_used_data":13440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327220,"statfs":{"total":96636764160,"available":96591073280,"internally_reserved":0,"allocated":13762560,"data_stored":15618609,"data_compressed":196617,"data_compressed_allocated":266240,"data_compressed_original":532480,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313233,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41460,"kb_used_data":14440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330380,"statfs":{"total":96636764160,"available":96594309120,"internally_reserved":0,"allocated":14786560,"data_stored":19677745,"data_compressed":196617,"data_compressed_allocated":266240,"data_compressed_original":532480,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,4,5,6,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313232,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37120,"kb_used_data":10092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334720,"statfs":{"total":96636764160,"available":96598753280,"internally_reserved":0,"allocated":10334208,"data_stored":11889168,"data_compressed":193535,"data_compressed_allocated":262144,"data_compressed_original":524288,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,4,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":0,"seq":257698037768,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313234,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45984,"kb_used_data":10640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94325856,"statfs":{"total":96636764160,"available":96589676544,"internally_reserved":0,"allocated":10895360,"data_stored":14942769,"data_compressed":196617,"data_compressed_allocated":266240,"data_compressed_original":532480,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[5,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313233,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40748,"kb_used_data":13728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331092,"statfs":{"total":96636764160,"available":96595038208,"internally_reserved":0,"allocated":14057472,"data_stored":15442481,"data_compressed":196617,"data_compressed_allocated":266240,"data_compressed_original":532480,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280530,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46308,"kb_used_data":15120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325532,"statfs":{"total":96636764160,"available":96589344768,"internally_reserved":0,"allocated":15482880,"data_stored":21416602,"data_compressed":199708,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17002496,"data_stored":22966272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14471168,"data_stored":20672512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15593472,"data_stored":17981440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2838528,"data_stored":4096000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15941632,"data_stored":19820544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11075584,"data_stored":14823424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15437824,"data_stored":19070976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12304384,"data_stored":16379904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12763136,"data_stored":14884864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":13787136,"data_stored":18944000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":9347072,"data_stored":11165696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4374528,"data_stored":4374528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1802,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":9895936,"data_stored":14209024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":12963840,"data_stored":16084992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":13058048,"data_stored":14708736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":9609216,"data_stored":11456512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:08:58.897 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-26T01:08:58.898 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-26T01:08:58.907 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:58.907 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:58.908 INFO:teuthology.orchestra.run.smithi012.stdout: "num_ops": 0 2024-04-26T01:08:58.908 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:58.912 INFO:teuthology.orchestra.run.smithi008.stdout:osd.8: {} 2024-04-26T01:08:58.913 INFO:teuthology.orchestra.run.smithi008.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:08:58.922 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:08:58.941 INFO:teuthology.orchestra.run.smithi008.stdout:osd.9: {} 2024-04-26T01:08:58.941 INFO:teuthology.orchestra.run.smithi008.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:08:58.969 INFO:teuthology.orchestra.run.smithi008.stdout:osd.10: {} 2024-04-26T01:08:58.969 INFO:teuthology.orchestra.run.smithi008.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:08:58.974 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:58.972+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:58.999 INFO:teuthology.orchestra.run.smithi008.stdout:osd.11: {} 2024-04-26T01:08:59.000 INFO:teuthology.orchestra.run.smithi008.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:08:59.027 INFO:teuthology.orchestra.run.smithi008.stdout:osd.12: {} 2024-04-26T01:08:59.027 INFO:teuthology.orchestra.run.smithi008.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:08:59.040 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:59.041 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:59.041 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:08:59.041 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:08:59.041 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:59.055 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:08:59.060 INFO:teuthology.orchestra.run.smithi008.stdout:osd.13: {} 2024-04-26T01:08:59.060 INFO:teuthology.orchestra.run.smithi008.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:08:59.075 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:59.072+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:59.087 INFO:teuthology.orchestra.run.smithi008.stdout:osd.14: {} 2024-04-26T01:08:59.087 INFO:teuthology.orchestra.run.smithi008.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:08:59.088 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:08:59.084+0000 7f50e18d3640 0 -- 172.21.15.8:0/3677311285 >> [v2:172.21.15.164:6803/2601606699,v1:172.21.15.164:6805/2601606699] conn(0x7f50dc223940 msgr2=0x7f50dc225d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-04-26T01:08:59.095 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-26T01:08:59.176 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:59.175+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:59.185 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:59.185 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:08:59.185 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:08:59.185 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:08:59.186 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.186 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.2s5 32/30 ECSubWrite(tid=157, reqid=client.4521.0:13, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:59.186 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.308314+0000", 2024-04-26T01:08:59.186 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 59.874957309999999, 2024-04-26T01:08:59.186 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011288362999999999, 2024-04-26T01:08:59.186 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.187 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.187 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.187 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.187 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.187 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.187 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308314+0000", 2024-04-26T01:08:59.187 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.187 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.187 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.187 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.188 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308314+0000", 2024-04-26T01:08:59.188 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.188 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.188 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.188 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308315+0000", 2024-04-26T01:08:59.188 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.6599999999999995e-07 2024-04-26T01:08:59.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.188 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.189 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.189 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308467+0000", 2024-04-26T01:08:59.189 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000152249 2024-04-26T01:08:59.189 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.189 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.189 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.190 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308602+0000", 2024-04-26T01:08:59.190 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000135048 2024-04-26T01:08:59.190 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.190 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.190 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.190 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308653+0000", 2024-04-26T01:08:59.191 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0864e-05 2024-04-26T01:08:59.191 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.191 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.191 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.191 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308834+0000", 2024-04-26T01:08:59.191 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000180466 2024-04-26T01:08:59.191 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.191 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.191 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.192 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.308857+0000", 2024-04-26T01:08:59.192 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2904e-05 2024-04-26T01:08:59.192 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.192 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.192 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.192 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.319506+0000", 2024-04-26T01:08:59.192 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010649141000000001 2024-04-26T01:08:59.192 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.192 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.192 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.192 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.319603+0000", 2024-04-26T01:08:59.192 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6924999999999998e-05 2024-04-26T01:08:59.192 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.193 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.193 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.193 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.193 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.193 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 32/30 ECSubWrite(tid=0, reqid=client.4521.0:16, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:59.193 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:59.193 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 59.872738990999999, 2024-04-26T01:08:59.193 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012119314000000001, 2024-04-26T01:08:59.193 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.193 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.193 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.193 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.193 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.194 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.194 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:59.194 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.194 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.194 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.194 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.194 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:59.194 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.194 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.194 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.194 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.194 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310533+0000", 2024-04-26T01:08:59.194 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.7000000000000004e-07 2024-04-26T01:08:59.195 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.195 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.195 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.195 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310733+0000", 2024-04-26T01:08:59.195 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019985599999999999 2024-04-26T01:08:59.195 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.195 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.195 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.195 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310792+0000", 2024-04-26T01:08:59.195 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.8894000000000002e-05 2024-04-26T01:08:59.195 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.195 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.195 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.195 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.310816+0000", 2024-04-26T01:08:59.196 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4094e-05 2024-04-26T01:08:59.196 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.196 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.196 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.196 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311022+0000", 2024-04-26T01:08:59.196 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00020608599999999999 2024-04-26T01:08:59.196 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.196 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.196 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.196 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311044+0000", 2024-04-26T01:08:59.197 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1889999999999999e-05 2024-04-26T01:08:59.197 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.197 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.197 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.197 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322407+0000", 2024-04-26T01:08:59.197 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011362778 2024-04-26T01:08:59.197 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.197 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.197 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.198 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322652+0000", 2024-04-26T01:08:59.198 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00024504600000000001 2024-04-26T01:08:59.198 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.198 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.198 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.198 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.198 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.198 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.es2 33/30 ECSubWrite(tid=0, reqid=client.4521.0:22, at_version=33'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:59.198 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.311344+0000", 2024-04-26T01:08:59.198 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 59.871927477, 2024-04-26T01:08:59.198 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010979113, 2024-04-26T01:08:59.198 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.198 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.199 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.199 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.199 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.199 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.199 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311344+0000", 2024-04-26T01:08:59.199 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.199 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.199 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.199 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.199 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311344+0000", 2024-04-26T01:08:59.200 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.200 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.200 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.200 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311351+0000", 2024-04-26T01:08:59.200 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1040000000000001e-06 2024-04-26T01:08:59.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.200 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.200 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.200 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311496+0000", 2024-04-26T01:08:59.200 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000144852 2024-04-26T01:08:59.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.200 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.201 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.201 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311639+0000", 2024-04-26T01:08:59.201 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014267800000000001 2024-04-26T01:08:59.201 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.201 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.201 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.201 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311663+0000", 2024-04-26T01:08:59.201 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4090000000000001e-05 2024-04-26T01:08:59.201 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.201 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.201 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.201 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311836+0000", 2024-04-26T01:08:59.201 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000173349 2024-04-26T01:08:59.201 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.202 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.202 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.202 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311859+0000", 2024-04-26T01:08:59.202 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2504e-05 2024-04-26T01:08:59.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.202 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.202 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.202 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322193+0000", 2024-04-26T01:08:59.202 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010334157 2024-04-26T01:08:59.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.202 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.202 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.202 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322323+0000", 2024-04-26T01:08:59.202 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013037899999999999 2024-04-26T01:08:59.203 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.203 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.203 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.203 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.203 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.203 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.8s3 32/30 ECSubWrite(tid=1, reqid=client.4521.0:8, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:59.203 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.312999+0000", 2024-04-26T01:08:59.203 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 59.870273015000002, 2024-04-26T01:08:59.203 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0094850960000000002, 2024-04-26T01:08:59.203 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.203 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.203 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.203 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.204 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.204 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.204 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312999+0000", 2024-04-26T01:08:59.204 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.204 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.204 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.204 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.204 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312999+0000", 2024-04-26T01:08:59.204 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.204 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.204 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.204 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.204 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313001+0000", 2024-04-26T01:08:59.204 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.21e-06 2024-04-26T01:08:59.204 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.205 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.205 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.205 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313076+0000", 2024-04-26T01:08:59.205 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.5525000000000006e-05 2024-04-26T01:08:59.205 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.205 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.205 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.205 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313194+0000", 2024-04-26T01:08:59.205 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000118004 2024-04-26T01:08:59.205 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.205 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.205 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.205 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313238+0000", 2024-04-26T01:08:59.205 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3523999999999999e-05 2024-04-26T01:08:59.206 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.206 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.206 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.206 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313570+0000", 2024-04-26T01:08:59.206 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033178100000000001 2024-04-26T01:08:59.206 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.206 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.206 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.206 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313595+0000", 2024-04-26T01:08:59.206 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5181000000000002e-05 2024-04-26T01:08:59.207 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.207 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.207 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.207 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322175+0000", 2024-04-26T01:08:59.207 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0085803070000000006 2024-04-26T01:08:59.207 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.207 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.207 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.207 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322484+0000", 2024-04-26T01:08:59.207 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000308564 2024-04-26T01:08:59.207 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.207 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.208 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.208 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.208 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.208 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.es2 33/30 ECSubWrite(tid=1, reqid=client.4521.0:23, at_version=33'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:08:59.208 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.313493+0000", 2024-04-26T01:08:59.208 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 59.869778982, 2024-04-26T01:08:59.208 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0089452720000000006, 2024-04-26T01:08:59.208 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.208 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.208 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.209 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.209 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.209 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.209 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313493+0000", 2024-04-26T01:08:59.209 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.209 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.209 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.209 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.209 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313493+0000", 2024-04-26T01:08:59.209 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.209 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.210 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.210 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.210 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313494+0000", 2024-04-26T01:08:59.210 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5039999999999999e-06 2024-04-26T01:08:59.210 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.210 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.210 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.210 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313593+0000", 2024-04-26T01:08:59.210 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.9355e-05 2024-04-26T01:08:59.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.211 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.211 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.211 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313756+0000", 2024-04-26T01:08:59.211 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000162762 2024-04-26T01:08:59.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.211 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.211 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.211 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313781+0000", 2024-04-26T01:08:59.211 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4893e-05 2024-04-26T01:08:59.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.211 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.211 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.211 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313944+0000", 2024-04-26T01:08:59.212 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016297600000000001 2024-04-26T01:08:59.212 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.212 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.212 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.212 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.313968+0000", 2024-04-26T01:08:59.212 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3986999999999999e-05 2024-04-26T01:08:59.212 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.212 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.212 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.212 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322340+0000", 2024-04-26T01:08:59.212 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0083720840000000001 2024-04-26T01:08:59.212 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.213 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.213 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.213 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.322438+0000", 2024-04-26T01:08:59.213 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7711e-05 2024-04-26T01:08:59.213 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.213 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.213 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.213 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.213 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.213 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.4s3 50/30 ECSubWrite(tid=29, reqid=osd.8.0:29, at_version=50'18, trim_to=0'0, roll_forward_to=47'16))", 2024-04-26T01:08:59.213 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.040723+0000", 2024-04-26T01:08:59.213 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 43.142548742000002, 2024-04-26T01:08:59.213 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012574701000000001, 2024-04-26T01:08:59.213 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.214 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.214 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.214 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.214 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.214 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.214 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040723+0000", 2024-04-26T01:08:59.214 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.214 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.214 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.214 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.214 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040723+0000", 2024-04-26T01:08:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040724+0000", 2024-04-26T01:08:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.37e-07 2024-04-26T01:08:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040778+0000", 2024-04-26T01:08:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.4453000000000002e-05 2024-04-26T01:08:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.216 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040860+0000", 2024-04-26T01:08:59.216 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.2060999999999999e-05 2024-04-26T01:08:59.216 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.216 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.216 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.216 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.040883+0000", 2024-04-26T01:08:59.216 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2994e-05 2024-04-26T01:08:59.216 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.216 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.216 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.216 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.041058+0000", 2024-04-26T01:08:59.216 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017496999999999999 2024-04-26T01:08:59.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.217 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.217 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.217 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.041082+0000", 2024-04-26T01:08:59.217 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.393e-05 2024-04-26T01:08:59.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.217 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.217 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.218 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053181+0000", 2024-04-26T01:08:59.218 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012099085000000001 2024-04-26T01:08:59.218 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.218 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.218 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.218 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053298+0000", 2024-04-26T01:08:59.218 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000116471 2024-04-26T01:08:59.218 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.219 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.219 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.219 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.219 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.219 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 50/30 ECSubWrite(tid=214, reqid=osd.4.0:214, at_version=50'22, trim_to=0'0, roll_forward_to=45'20))", 2024-04-26T01:08:59.219 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.047877+0000", 2024-04-26T01:08:59.219 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 43.135394077999997, 2024-04-26T01:08:59.219 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010720746, 2024-04-26T01:08:59.220 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.220 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.220 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.220 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.220 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.220 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.220 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047877+0000", 2024-04-26T01:08:59.220 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.220 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.221 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.221 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.221 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047877+0000", 2024-04-26T01:08:59.221 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.221 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.221 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.221 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047878+0000", 2024-04-26T01:08:59.222 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.6699999999999999e-07 2024-04-26T01:08:59.222 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.222 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.222 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.222 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.047930+0000", 2024-04-26T01:08:59.222 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.1669999999999998e-05 2024-04-26T01:08:59.222 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.222 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.223 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.223 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048154+0000", 2024-04-26T01:08:59.223 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022397399999999999 2024-04-26T01:08:59.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.223 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.223 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.223 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048176+0000", 2024-04-26T01:08:59.223 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2734e-05 2024-04-26T01:08:59.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.224 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.224 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.224 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048452+0000", 2024-04-26T01:08:59.224 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00027581799999999999 2024-04-26T01:08:59.224 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.224 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.224 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.224 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.048478+0000", 2024-04-26T01:08:59.225 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5723000000000001e-05 2024-04-26T01:08:59.225 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.225 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.225 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.225 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058506+0000", 2024-04-26T01:08:59.225 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010027635999999999 2024-04-26T01:08:59.225 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.225 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.225 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.226 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.058598+0000", 2024-04-26T01:08:59.226 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2724000000000002e-05 2024-04-26T01:08:59.226 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.226 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.226 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.226 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.226 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.226 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.6s2 50/30 ECSubWrite(tid=41, reqid=osd.0.0:41, at_version=50'24, trim_to=0'0, roll_forward_to=45'22))", 2024-04-26T01:08:59.227 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.052765+0000", 2024-04-26T01:08:59.227 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 43.130506334000003, 2024-04-26T01:08:59.227 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0080835920000000006, 2024-04-26T01:08:59.227 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.227 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.227 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.227 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.227 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.228 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.228 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052765+0000", 2024-04-26T01:08:59.228 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.228 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.228 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.228 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052765+0000", 2024-04-26T01:08:59.228 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.229 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.229 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.229 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052766+0000", 2024-04-26T01:08:59.229 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1999999999999999e-07 2024-04-26T01:08:59.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.229 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.229 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.229 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052820+0000", 2024-04-26T01:08:59.230 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.4441000000000003e-05 2024-04-26T01:08:59.230 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.230 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.230 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.230 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052916+0000", 2024-04-26T01:08:59.230 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5597e-05 2024-04-26T01:08:59.230 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.230 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.231 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.231 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052940+0000", 2024-04-26T01:08:59.231 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3626e-05 2024-04-26T01:08:59.231 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.231 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.231 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.231 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053100+0000", 2024-04-26T01:08:59.231 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015994800000000001 2024-04-26T01:08:59.231 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.231 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.231 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.231 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053126+0000", 2024-04-26T01:08:59.231 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6393e-05 2024-04-26T01:08:59.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.232 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.232 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.232 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060748+0000", 2024-04-26T01:08:59.232 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.007622523 2024-04-26T01:08:59.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.232 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.232 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.232 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.060849+0000", 2024-04-26T01:08:59.232 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000100344 2024-04-26T01:08:59.232 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.232 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.233 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.233 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.233 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 55/30 ECSubWrite(tid=254, reqid=client.4521.0:549, at_version=55'29, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:59.233 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:21.088071+0000", 2024-04-26T01:08:59.233 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 38.095200140000003, 2024-04-26T01:08:59.233 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.023607655000000002, 2024-04-26T01:08:59.233 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.233 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.233 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.233 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.233 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.233 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.233 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.088071+0000", 2024-04-26T01:08:59.234 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.234 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.234 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.234 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.234 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.088071+0000", 2024-04-26T01:08:59.234 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.234 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.234 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.234 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.234 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.088074+0000", 2024-04-26T01:08:59.234 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.097e-06 2024-04-26T01:08:59.234 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.234 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.235 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.235 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.088360+0000", 2024-04-26T01:08:59.235 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000286261 2024-04-26T01:08:59.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.235 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.235 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.236 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.103040+0000", 2024-04-26T01:08:59.236 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.014680442 2024-04-26T01:08:59.236 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.236 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.236 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.236 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.103049+0000", 2024-04-26T01:08:59.236 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.251e-06 2024-04-26T01:08:59.236 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.236 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.236 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.237 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.103185+0000", 2024-04-26T01:08:59.237 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013595400000000001 2024-04-26T01:08:59.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.237 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.237 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.237 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.103212+0000", 2024-04-26T01:08:59.237 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6517e-05 2024-04-26T01:08:59.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.237 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.237 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.237 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.111554+0000", 2024-04-26T01:08:59.237 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.008342426 2024-04-26T01:08:59.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.238 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.238 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.238 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.111679+0000", 2024-04-26T01:08:59.238 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000124707 2024-04-26T01:08:59.238 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.238 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.238 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.238 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.238 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.239 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 55/30 ECSubWrite(tid=256, reqid=client.4521.0:550, at_version=55'30, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:08:59.239 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:21.102883+0000", 2024-04-26T01:08:59.239 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 38.080388286999998, 2024-04-26T01:08:59.239 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0094720329999999995, 2024-04-26T01:08:59.239 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.239 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.239 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.239 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.239 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.239 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.239 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.102883+0000", 2024-04-26T01:08:59.239 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.240 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.240 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.240 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.102883+0000", 2024-04-26T01:08:59.240 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.240 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.240 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.240 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.102884+0000", 2024-04-26T01:08:59.240 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8000000000000005e-07 2024-04-26T01:08:59.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.240 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.240 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.103016+0000", 2024-04-26T01:08:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013183699999999999 2024-04-26T01:08:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105250+0000", 2024-04-26T01:08:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002234307 2024-04-26T01:08:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105272+0000", 2024-04-26T01:08:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1886e-05 2024-04-26T01:08:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105352+0000", 2024-04-26T01:08:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.0383999999999995e-05 2024-04-26T01:08:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.105364+0000", 2024-04-26T01:08:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1331999999999999e-05 2024-04-26T01:08:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.112272+0000", 2024-04-26T01:08:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0069084970000000004 2024-04-26T01:08:59.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.243 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.243 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.243 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:21.112355+0000", 2024-04-26T01:08:59.243 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.3010000000000007e-05 2024-04-26T01:08:59.243 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.243 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.243 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.243 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.243 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 60/30 ECSubWrite(tid=351, reqid=osd.4.0:351, at_version=60'54, trim_to=0'0, roll_forward_to=59'50))", 2024-04-26T01:08:59.244 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:26.179502+0000", 2024-04-26T01:08:59.244 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 33.003769542999997, 2024-04-26T01:08:59.244 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.009331499, 2024-04-26T01:08:59.244 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.244 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.244 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.245 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.245 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.245 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.245 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179502+0000", 2024-04-26T01:08:59.245 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.245 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179502+0000", 2024-04-26T01:08:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179502+0000", 2024-04-26T01:08:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.7e-07 2024-04-26T01:08:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179532+0000", 2024-04-26T01:08:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9977000000000001e-05 2024-04-26T01:08:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179726+0000", 2024-04-26T01:08:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019319300000000001 2024-04-26T01:08:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.179745+0000", 2024-04-26T01:08:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9167000000000001e-05 2024-04-26T01:08:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.183638+0000", 2024-04-26T01:08:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0038929580000000002 2024-04-26T01:08:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.183663+0000", 2024-04-26T01:08:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5673999999999998e-05 2024-04-26T01:08:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.188734+0000", 2024-04-26T01:08:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0050702530000000003 2024-04-26T01:08:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:26.188834+0000", 2024-04-26T01:08:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.9807000000000002e-05 2024-04-26T01:08:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.6s2 81/30 ECSubWrite(tid=96, reqid=client.4521.0:892, at_version=81'67, trim_to=0'0, roll_forward_to=81'65))", 2024-04-26T01:08:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:47.479409+0000", 2024-04-26T01:08:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 11.70386289, 2024-04-26T01:08:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0089288479999999996, 2024-04-26T01:08:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.479409+0000", 2024-04-26T01:08:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.479409+0000", 2024-04-26T01:08:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.479410+0000", 2024-04-26T01:08:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7740000000000001e-06 2024-04-26T01:08:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.479454+0000", 2024-04-26T01:08:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3713e-05 2024-04-26T01:08:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.479543+0000", 2024-04-26T01:08:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.8370999999999998e-05 2024-04-26T01:08:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.479558+0000", 2024-04-26T01:08:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5707000000000001e-05 2024-04-26T01:08:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.479722+0000", 2024-04-26T01:08:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016331399999999999 2024-04-26T01:08:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.479748+0000", 2024-04-26T01:08:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5950000000000001e-05 2024-04-26T01:08:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488235+0000", 2024-04-26T01:08:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0084874649999999996 2024-04-26T01:08:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488338+0000", 2024-04-26T01:08:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00010255399999999999 2024-04-26T01:08:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.6s2 81/30 ECSubWrite(tid=97, reqid=client.4521.0:893, at_version=81'68, trim_to=0'0, roll_forward_to=81'65))", 2024-04-26T01:08:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:47.481020+0000", 2024-04-26T01:08:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 11.702251835, 2024-04-26T01:08:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011366071, 2024-04-26T01:08:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.481020+0000", 2024-04-26T01:08:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.481020+0000", 2024-04-26T01:08:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.481021+0000", 2024-04-26T01:08:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.573e-06 2024-04-26T01:08:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.481068+0000", 2024-04-26T01:08:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.6366999999999998e-05 2024-04-26T01:08:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.481148+0000", 2024-04-26T01:08:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.0433999999999996e-05 2024-04-26T01:08:59.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.258 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.258 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.258 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.481169+0000", 2024-04-26T01:08:59.258 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0716999999999998e-05 2024-04-26T01:08:59.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.258 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.258 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.258 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.481436+0000", 2024-04-26T01:08:59.258 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00026681799999999999 2024-04-26T01:08:59.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.258 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.258 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.259 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.481472+0000", 2024-04-26T01:08:59.259 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.6143999999999998e-05 2024-04-26T01:08:59.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.259 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.259 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.259 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.492284+0000", 2024-04-26T01:08:59.259 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010812554 2024-04-26T01:08:59.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.259 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.259 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.259 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.492386+0000", 2024-04-26T01:08:59.259 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000101464 2024-04-26T01:08:59.259 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.259 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.260 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.260 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.6s2 81/30 ECSubWrite(tid=98, reqid=client.4521.0:894, at_version=81'69, trim_to=0'0, roll_forward_to=81'65))", 2024-04-26T01:08:59.260 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:47.482448+0000", 2024-04-26T01:08:59.260 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 11.700823477, 2024-04-26T01:08:59.260 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.01000564, 2024-04-26T01:08:59.260 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.260 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.260 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.260 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.260 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.260 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482448+0000", 2024-04-26T01:08:59.260 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.261 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.261 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482448+0000", 2024-04-26T01:08:59.261 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.261 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.261 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482450+0000", 2024-04-26T01:08:59.261 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.703e-06 2024-04-26T01:08:59.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.261 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.261 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482493+0000", 2024-04-26T01:08:59.262 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3146999999999999e-05 2024-04-26T01:08:59.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.262 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.262 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.262 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482593+0000", 2024-04-26T01:08:59.262 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00010004400000000001 2024-04-26T01:08:59.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.262 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.262 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.262 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482616+0000", 2024-04-26T01:08:59.262 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2583999999999999e-05 2024-04-26T01:08:59.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.262 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.263 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.263 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.483204+0000", 2024-04-26T01:08:59.263 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00058860900000000003 2024-04-26T01:08:59.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.263 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.263 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.263 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.483230+0000", 2024-04-26T01:08:59.263 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5590000000000001e-05 2024-04-26T01:08:59.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.263 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.263 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.263 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.492395+0000", 2024-04-26T01:08:59.263 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0091650789999999996 2024-04-26T01:08:59.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.263 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.264 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.264 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.492454+0000", 2024-04-26T01:08:59.264 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.8884000000000001e-05 2024-04-26T01:08:59.264 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.264 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.264 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.264 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.264 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 81/30 ECSubWrite(tid=476, reqid=client.4521.0:896, at_version=81'69, trim_to=0'0, roll_forward_to=81'67))", 2024-04-26T01:08:59.264 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:47.484152+0000", 2024-04-26T01:08:59.264 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 11.699119161, 2024-04-26T01:08:59.264 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012034947000000001, 2024-04-26T01:08:59.264 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.264 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.265 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.265 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.265 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.265 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.265 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.484152+0000", 2024-04-26T01:08:59.265 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.265 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.265 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.265 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.484152+0000", 2024-04-26T01:08:59.265 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.265 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.265 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.265 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.484154+0000", 2024-04-26T01:08:59.266 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9800000000000001e-06 2024-04-26T01:08:59.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.266 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.266 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.266 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.484210+0000", 2024-04-26T01:08:59.266 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.6002999999999999e-05 2024-04-26T01:08:59.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.266 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.266 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.266 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488357+0000", 2024-04-26T01:08:59.266 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0041462959999999998 2024-04-26T01:08:59.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.266 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.267 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.267 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488380+0000", 2024-04-26T01:08:59.267 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3743999999999998e-05 2024-04-26T01:08:59.267 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.267 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.267 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.267 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488555+0000", 2024-04-26T01:08:59.268 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000174737 2024-04-26T01:08:59.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.268 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.268 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.268 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488585+0000", 2024-04-26T01:08:59.268 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.951e-05 2024-04-26T01:08:59.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.268 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.268 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.268 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.496092+0000", 2024-04-26T01:08:59.268 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0075069890000000004 2024-04-26T01:08:59.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.269 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.269 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.269 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.496187+0000", 2024-04-26T01:08:59.269 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5687999999999996e-05 2024-04-26T01:08:59.269 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.269 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.269 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.269 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.269 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 81/30 ECSubWrite(tid=478, reqid=client.4521.0:897, at_version=81'70, trim_to=0'0, roll_forward_to=81'67))", 2024-04-26T01:08:59.269 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:47.488186+0000", 2024-04-26T01:08:59.269 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 11.695085316, 2024-04-26T01:08:59.269 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012223111, 2024-04-26T01:08:59.270 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.270 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.270 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.270 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.270 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.270 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.270 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488186+0000", 2024-04-26T01:08:59.270 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.270 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.270 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.270 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.270 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488186+0000", 2024-04-26T01:08:59.271 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.271 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.271 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488188+0000", 2024-04-26T01:08:59.271 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0030000000000001e-06 2024-04-26T01:08:59.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.271 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.271 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488237+0000", 2024-04-26T01:08:59.271 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.914e-05 2024-04-26T01:08:59.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.272 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.272 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.272 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488704+0000", 2024-04-26T01:08:59.273 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000466909 2024-04-26T01:08:59.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.273 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.273 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.273 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488724+0000", 2024-04-26T01:08:59.273 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9463999999999999e-05 2024-04-26T01:08:59.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.273 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.273 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.273 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.492485+0000", 2024-04-26T01:08:59.273 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0037614229999999998 2024-04-26T01:08:59.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.274 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.274 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.274 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.492510+0000", 2024-04-26T01:08:59.274 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5066e-05 2024-04-26T01:08:59.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.275 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.275 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.275 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.500255+0000", 2024-04-26T01:08:59.275 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0077452409999999999 2024-04-26T01:08:59.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.275 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.275 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.275 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.500409+0000", 2024-04-26T01:08:59.275 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015386499999999999 2024-04-26T01:08:59.275 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.275 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.275 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.275 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.276 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 81/30 ECSubWrite(tid=480, reqid=client.4521.0:898, at_version=81'71, trim_to=0'0, roll_forward_to=81'67))", 2024-04-26T01:08:59.276 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:47.488247+0000", 2024-04-26T01:08:59.276 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 11.695024375999999, 2024-04-26T01:08:59.276 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012226287000000001, 2024-04-26T01:08:59.276 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.276 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.276 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.276 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.276 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.276 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.276 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488247+0000", 2024-04-26T01:08:59.276 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.276 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488247+0000", 2024-04-26T01:08:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488248+0000", 2024-04-26T01:08:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.7000000000000004e-07 2024-04-26T01:08:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488284+0000", 2024-04-26T01:08:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.6213e-05 2024-04-26T01:08:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.278 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.278 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.278 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488831+0000", 2024-04-26T01:08:59.278 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00054683300000000004 2024-04-26T01:08:59.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.278 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.278 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.278 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.488848+0000", 2024-04-26T01:08:59.278 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6739999999999999e-05 2024-04-26T01:08:59.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.278 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.278 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.278 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.494813+0000", 2024-04-26T01:08:59.278 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0059649450000000001 2024-04-26T01:08:59.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.279 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.279 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.279 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.494834+0000", 2024-04-26T01:08:59.279 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1080000000000001e-05 2024-04-26T01:08:59.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.279 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.279 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.279 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.500417+0000", 2024-04-26T01:08:59.279 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0055835260000000001 2024-04-26T01:08:59.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.279 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.280 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.280 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.500473+0000", 2024-04-26T01:08:59.280 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.6280000000000003e-05 2024-04-26T01:08:59.280 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.280 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.280 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.280 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.280 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.280 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 83/30 ECSubWrite(tid=495, reqid=osd.4.0:495, at_version=83'75, trim_to=0'0, roll_forward_to=82'73))", 2024-04-26T01:08:59.280 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:49.329279+0000", 2024-04-26T01:08:59.280 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.8539929589999993, 2024-04-26T01:08:59.280 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0080367549999999996, 2024-04-26T01:08:59.280 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.280 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.281 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.281 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.281 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.281 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.281 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.329279+0000", 2024-04-26T01:08:59.281 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.281 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.281 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.281 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.282 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.329279+0000", 2024-04-26T01:08:59.282 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.282 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.282 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.282 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.329279+0000", 2024-04-26T01:08:59.282 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.3000000000000001e-07 2024-04-26T01:08:59.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.282 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.282 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.282 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.329334+0000", 2024-04-26T01:08:59.282 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.5040999999999997e-05 2024-04-26T01:08:59.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.282 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.283 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.283 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.329681+0000", 2024-04-26T01:08:59.283 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00034688399999999999 2024-04-26T01:08:59.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.283 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.283 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.283 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.329705+0000", 2024-04-26T01:08:59.283 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3683e-05 2024-04-26T01:08:59.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.283 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.283 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.283 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.329978+0000", 2024-04-26T01:08:59.283 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00027329200000000001 2024-04-26T01:08:59.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.283 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.284 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.284 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.330006+0000", 2024-04-26T01:08:59.284 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8416999999999999e-05 2024-04-26T01:08:59.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.284 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.284 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.284 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.337204+0000", 2024-04-26T01:08:59.284 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0071976640000000003 2024-04-26T01:08:59.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.284 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.284 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.284 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.337315+0000", 2024-04-26T01:08:59.284 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00011124399999999999 2024-04-26T01:08:59.284 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.285 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.285 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.285 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.285 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.285 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs2 83/30 ECSubWrite(tid=497, reqid=osd.4.0:497, at_version=83'77, trim_to=0'0, roll_forward_to=82'73))", 2024-04-26T01:08:59.285 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:49.332263+0000", 2024-04-26T01:08:59.285 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 9.8510082029999992, 2024-04-26T01:08:59.286 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010570816, 2024-04-26T01:08:59.286 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.286 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.286 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.286 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.286 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.287 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.287 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.332263+0000", 2024-04-26T01:08:59.287 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.287 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.287 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.287 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.332263+0000", 2024-04-26T01:08:59.287 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.287 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.288 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.288 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.332265+0000", 2024-04-26T01:08:59.288 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7400000000000001e-06 2024-04-26T01:08:59.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.288 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.288 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.288 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.332318+0000", 2024-04-26T01:08:59.288 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.3050999999999997e-05 2024-04-26T01:08:59.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.288 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.288 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.288 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.332407+0000", 2024-04-26T01:08:59.288 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.8560000000000006e-05 2024-04-26T01:08:59.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.289 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.289 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.289 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.332428+0000", 2024-04-26T01:08:59.289 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1662999999999999e-05 2024-04-26T01:08:59.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.289 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.289 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.290 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.332581+0000", 2024-04-26T01:08:59.290 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015222099999999999 2024-04-26T01:08:59.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.290 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.290 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.290 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.332607+0000", 2024-04-26T01:08:59.290 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6186999999999998e-05 2024-04-26T01:08:59.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.291 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.291 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.291 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.342740+0000", 2024-04-26T01:08:59.291 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010133329999999999 2024-04-26T01:08:59.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.291 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.291 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.291 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:49.342834+0000", 2024-04-26T01:08:59.291 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.4063999999999999e-05 2024-04-26T01:08:59.291 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.292 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.292 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.292 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.292 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.292 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s3 88/30 ECSubWrite(tid=108, reqid=client.4521.0:947, at_version=88'82, trim_to=0'0, roll_forward_to=81'80))", 2024-04-26T01:08:59.292 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:54.063070+0000", 2024-04-26T01:08:59.292 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 5.1202018159999998, 2024-04-26T01:08:59.292 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0092957330000000005, 2024-04-26T01:08:59.292 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.292 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.292 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.293 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.293 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.293 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.293 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.063070+0000", 2024-04-26T01:08:59.293 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.293 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.293 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.293 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.294 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.063070+0000", 2024-04-26T01:08:59.294 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.294 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.294 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.294 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.294 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.063070+0000", 2024-04-26T01:08:59.294 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.46e-07 2024-04-26T01:08:59.294 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.294 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.294 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.294 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.063114+0000", 2024-04-26T01:08:59.294 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3866999999999999e-05 2024-04-26T01:08:59.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.295 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.295 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.295 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.063217+0000", 2024-04-26T01:08:59.295 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00010305 2024-04-26T01:08:59.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.295 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.295 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.295 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.063243+0000", 2024-04-26T01:08:59.295 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6239999999999999e-05 2024-04-26T01:08:59.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.296 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.296 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.296 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.063412+0000", 2024-04-26T01:08:59.296 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000168531 2024-04-26T01:08:59.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.296 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.296 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.296 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.063440+0000", 2024-04-26T01:08:59.296 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8422999999999999e-05 2024-04-26T01:08:59.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.296 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.296 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.297 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.072299+0000", 2024-04-26T01:08:59.297 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0088584960000000004 2024-04-26T01:08:59.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.297 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.297 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.297 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.072365+0000", 2024-04-26T01:08:59.297 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.6680000000000005e-05 2024-04-26T01:08:59.297 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.297 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.297 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.297 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.297 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.297 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:59.299 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:08:59.299 INFO:tasks.ceph.osd.14.smithi116.stderr:2024-04-26T01:08:59.295+0000 7f7d63522640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25060) UID: 0 2024-04-26T01:08:59.321 INFO:teuthology.orchestra.run.smithi008.stdout:osd.15: {} 2024-04-26T01:08:59.321 INFO:teuthology.orchestra.run.smithi008.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:08:59.325 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T01:08:59.326 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T01:08:59.399 INFO:tasks.ceph.osd.5.smithi012.stderr:2024-04-26T01:08:59.397+0000 7f6ad17db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24942) UID: 0 2024-04-26T01:08:59.430 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:59.430 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:59.430 INFO:teuthology.orchestra.run.smithi012.stdout: "num_ops": 0 2024-04-26T01:08:59.431 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:59.445 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:08:59.500 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:59.496+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:59.601 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:08:59.600+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:08:59.611 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:59.611 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:08:59.611 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:08:59.611 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:08:59.611 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:59.626 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:08:59.702 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-04-26T01:08:59.702+0000 7f7180d68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25266) UID: 0 2024-04-26T01:08:59.750 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:08:59.751 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:08:59.751 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:08:59.751 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:08:59.751 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.751 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:7 3.8s0 3:1b6b0095:::smithi16428812-3:head [append 0~49152,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:59.751 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.302696+0000", 2024-04-26T01:08:59.751 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 60.446328966999999, 2024-04-26T01:08:59.751 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.016152680999999999, 2024-04-26T01:08:59.751 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.751 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.751 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:59.751 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 7 2024-04-26T01:08:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.302696+0000", 2024-04-26T01:08:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.302696+0000", 2024-04-26T01:08:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.753 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.753 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.302704+0000", 2024-04-26T01:08:59.753 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1130000000000001e-06 2024-04-26T01:08:59.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.753 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.753 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303173+0000", 2024-04-26T01:08:59.754 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000469109 2024-04-26T01:08:59.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.754 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.754 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.754 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303207+0000", 2024-04-26T01:08:59.754 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.3937999999999999e-05 2024-04-26T01:08:59.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.754 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.754 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.754 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303229+0000", 2024-04-26T01:08:59.754 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1573000000000002e-05 2024-04-26T01:08:59.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.755 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.755 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.755 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304061+0000", 2024-04-26T01:08:59.755 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00083202500000000002 2024-04-26T01:08:59.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.755 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.755 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:59.755 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.304318+0000", 2024-04-26T01:08:59.755 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025779299999999998 2024-04-26T01:08:59.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.755 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.755 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.755 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.306191+0000", 2024-04-26T01:08:59.755 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0018724010000000001 2024-04-26T01:08:59.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.756 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.756 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.756 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318520+0000", 2024-04-26T01:08:59.756 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012329028000000001 2024-04-26T01:08:59.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.756 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.756 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:59.756 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318628+0000", 2024-04-26T01:08:59.756 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00010778800000000001 2024-04-26T01:08:59.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.756 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.756 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.756 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318849+0000", 2024-04-26T01:08:59.756 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022091299999999999 2024-04-26T01:08:59.757 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.757 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.757 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.757 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.757 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.757 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:8 3.8s0 3:1b6b0095:::smithi16428812-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:59.757 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.303305+0000", 2024-04-26T01:08:59.757 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 60.445719676000003, 2024-04-26T01:08:59.757 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.020292667, 2024-04-26T01:08:59.757 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.757 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.757 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:59.758 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:59.758 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:59.758 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:59.758 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 8 2024-04-26T01:08:59.758 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.758 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.758 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.758 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.758 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303305+0000", 2024-04-26T01:08:59.758 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.758 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.758 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.758 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.759 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303305+0000", 2024-04-26T01:08:59.759 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.759 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.759 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.759 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.759 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303307+0000", 2024-04-26T01:08:59.759 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7770000000000001e-06 2024-04-26T01:08:59.759 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.759 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.759 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.759 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303320+0000", 2024-04-26T01:08:59.759 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3086e-05 2024-04-26T01:08:59.759 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.759 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.760 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.760 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303343+0000", 2024-04-26T01:08:59.760 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3323999999999999e-05 2024-04-26T01:08:59.760 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.760 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.760 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.760 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303387+0000", 2024-04-26T01:08:59.760 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3741e-05 2024-04-26T01:08:59.760 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.760 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.760 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.760 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.307388+0000", 2024-04-26T01:08:59.760 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0040010430000000001 2024-04-26T01:08:59.760 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.761 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.761 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:59.761 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.307645+0000", 2024-04-26T01:08:59.761 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025681400000000001 2024-04-26T01:08:59.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.761 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.761 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.761 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.311893+0000", 2024-04-26T01:08:59.761 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.004247626 2024-04-26T01:08:59.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.761 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.761 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.761 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.318857+0000", 2024-04-26T01:08:59.761 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0069646760000000004 2024-04-26T01:08:59.762 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.762 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.762 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:59.762 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323372+0000", 2024-04-26T01:08:59.762 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0045143329999999997 2024-04-26T01:08:59.762 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.762 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.762 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.762 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.323598+0000", 2024-04-26T01:08:59.762 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000226247 2024-04-26T01:08:59.762 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.762 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.762 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.763 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.763 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.763 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:9 3.8s0 3:1b6b0095:::smithi16428812-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:08:59.763 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.303410+0000", 2024-04-26T01:08:59.763 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 60.445614595000002, 2024-04-26T01:08:59.763 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.028697769000000001, 2024-04-26T01:08:59.763 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.763 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.763 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:59.763 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:59.763 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:59.763 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:59.763 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 9 2024-04-26T01:08:59.763 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.763 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.764 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.764 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.764 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303410+0000", 2024-04-26T01:08:59.764 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.764 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.764 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.764 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.764 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303410+0000", 2024-04-26T01:08:59.764 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.764 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.764 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.764 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.764 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303412+0000", 2024-04-26T01:08:59.765 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.137e-06 2024-04-26T01:08:59.765 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.765 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.765 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.765 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303420+0000", 2024-04-26T01:08:59.765 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.3540000000000002e-06 2024-04-26T01:08:59.765 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.765 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.765 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.765 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303439+0000", 2024-04-26T01:08:59.765 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9227000000000001e-05 2024-04-26T01:08:59.765 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.766 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.766 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.766 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.303455+0000", 2024-04-26T01:08:59.766 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5852999999999999e-05 2024-04-26T01:08:59.766 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.766 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.766 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.767 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312566+0000", 2024-04-26T01:08:59.767 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0091115209999999992 2024-04-26T01:08:59.767 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.767 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.767 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:59.767 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.312793+0000", 2024-04-26T01:08:59.767 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022636999999999999 2024-04-26T01:08:59.768 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.768 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.768 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.768 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.325814+0000", 2024-04-26T01:08:59.768 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013020893 2024-04-26T01:08:59.768 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.769 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.769 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:59.769 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.326042+0000", 2024-04-26T01:08:59.769 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022811899999999999 2024-04-26T01:08:59.769 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.769 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.770 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.770 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.332108+0000", 2024-04-26T01:08:59.770 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0060662950000000002 2024-04-26T01:08:59.770 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.770 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.770 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.770 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.771 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.771 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:50 3.8s0 3:11b43273:::smithi16428812-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:59.771 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.344014+0000", 2024-04-26T01:08:59.771 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 60.405010896, 2024-04-26T01:08:59.771 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013017295999999999, 2024-04-26T01:08:59.771 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.772 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.772 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:59.772 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:59.772 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:59.772 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:59.772 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 50 2024-04-26T01:08:59.773 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.773 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.773 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.773 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.773 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344014+0000", 2024-04-26T01:08:59.773 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.774 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.774 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.774 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.774 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344014+0000", 2024-04-26T01:08:59.774 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.774 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.775 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.775 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.775 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344015+0000", 2024-04-26T01:08:59.775 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.437e-06 2024-04-26T01:08:59.775 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.775 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.775 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.775 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344022+0000", 2024-04-26T01:08:59.775 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.9970000000000001e-06 2024-04-26T01:08:59.775 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.776 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.776 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.776 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344034+0000", 2024-04-26T01:08:59.776 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1817000000000001e-05 2024-04-26T01:08:59.776 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.776 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.777 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.777 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344043+0000", 2024-04-26T01:08:59.777 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.8060000000000001e-06 2024-04-26T01:08:59.777 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.777 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.777 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.778 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.347851+0000", 2024-04-26T01:08:59.778 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0038083570000000001 2024-04-26T01:08:59.778 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.778 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.778 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:59.778 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.348142+0000", 2024-04-26T01:08:59.778 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00029042400000000001 2024-04-26T01:08:59.779 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.779 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.779 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.779 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.350786+0000", 2024-04-26T01:08:59.779 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002644202 2024-04-26T01:08:59.779 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.779 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.779 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.779 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.352937+0000", 2024-04-26T01:08:59.780 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0021514910000000002 2024-04-26T01:08:59.780 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.780 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.780 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:59.780 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.356940+0000", 2024-04-26T01:08:59.780 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0040022740000000001 2024-04-26T01:08:59.780 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.780 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.780 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.780 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357031+0000", 2024-04-26T01:08:59.781 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.1490999999999995e-05 2024-04-26T01:08:59.781 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.781 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.781 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.781 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.781 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.781 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:51 3.8s0 3:11b43273:::smithi16428812-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:08:59.781 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.344056+0000", 2024-04-26T01:08:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 60.404968885000002, 2024-04-26T01:08:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.01557094, 2024-04-26T01:08:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 51 2024-04-26T01:08:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344056+0000", 2024-04-26T01:08:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344056+0000", 2024-04-26T01:08:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344057+0000", 2024-04-26T01:08:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1699999999999997e-07 2024-04-26T01:08:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344060+0000", 2024-04-26T01:08:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.703e-06 2024-04-26T01:08:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344071+0000", 2024-04-26T01:08:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.057e-05 2024-04-26T01:08:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.344085+0000", 2024-04-26T01:08:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3997e-05 2024-04-26T01:08:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.351519+0000", 2024-04-26T01:08:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0074343079999999997 2024-04-26T01:08:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:08:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.351770+0000", 2024-04-26T01:08:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025095500000000003 2024-04-26T01:08:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357516+0000", 2024-04-26T01:08:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0057452919999999999 2024-04-26T01:08:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357612+0000", 2024-04-26T01:08:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6444e-05 2024-04-26T01:08:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.359627+0000", 2024-04-26T01:08:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.002014854 2024-04-26T01:08:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 38/30 ECSubWrite(tid=85, reqid=osd.4.0:85, at_version=38'19, trim_to=0'0, roll_forward_to=38'15))", 2024-04-26T01:08:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.432265+0000", 2024-04-26T01:08:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 55.316759914999999, 2024-04-26T01:08:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011946251999999999, 2024-04-26T01:08:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432265+0000", 2024-04-26T01:08:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432265+0000", 2024-04-26T01:08:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432266+0000", 2024-04-26T01:08:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5900000000000005e-07 2024-04-26T01:08:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.432295+0000", 2024-04-26T01:08:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9374999999999999e-05 2024-04-26T01:08:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441494+0000", 2024-04-26T01:08:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0091991369999999996 2024-04-26T01:08:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441529+0000", 2024-04-26T01:08:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.4677e-05 2024-04-26T01:08:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441702+0000", 2024-04-26T01:08:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017256500000000001 2024-04-26T01:08:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.441731+0000", 2024-04-26T01:08:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8897000000000001e-05 2024-04-26T01:08:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.444168+0000", 2024-04-26T01:08:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0024374930000000002 2024-04-26T01:08:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.444211+0000", 2024-04-26T01:08:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3149000000000003e-05 2024-04-26T01:08:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 38/30 ECSubWrite(tid=87, reqid=client.4521.0:267, at_version=0'0, trim_to=0'0, roll_forward_to=38'21))", 2024-04-26T01:08:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.447653+0000", 2024-04-26T01:08:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 55.301371515, 2024-04-26T01:08:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.55546236199999999, 2024-04-26T01:08:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.795 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.795 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.795 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.795 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.795 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.795 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.447653+0000", 2024-04-26T01:08:59.795 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.795 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.795 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.795 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.447653+0000", 2024-04-26T01:08:59.795 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.795 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.796 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.796 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.447654+0000", 2024-04-26T01:08:59.796 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.8299999999999996e-07 2024-04-26T01:08:59.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.796 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.796 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.796 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.447665+0000", 2024-04-26T01:08:59.796 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0539e-05 2024-04-26T01:08:59.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.796 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.796 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.796 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.001792+0000", 2024-04-26T01:08:59.796 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.55412781 2024-04-26T01:08:59.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.796 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.797 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.797 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.001811+0000", 2024-04-26T01:08:59.797 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8236999999999998e-05 2024-04-26T01:08:59.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.797 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.797 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.797 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.001927+0000", 2024-04-26T01:08:59.797 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000116582 2024-04-26T01:08:59.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.798 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.798 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.798 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.001939+0000", 2024-04-26T01:08:59.798 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2312e-05 2024-04-26T01:08:59.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.798 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.798 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.799 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.003044+0000", 2024-04-26T01:08:59.799 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0011047819999999999 2024-04-26T01:08:59.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.799 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.799 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.003116+0000", 2024-04-26T01:08:59.799 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1416999999999997e-05 2024-04-26T01:08:59.799 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.799 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.799 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.799 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/44 ECSubWrite(tid=215, reqid=client.4521.0:490, at_version=50'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-26T01:08:59.799 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.059123+0000", 2024-04-26T01:08:59.800 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 43.689901601999999, 2024-04-26T01:08:59.800 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.014980383999999999, 2024-04-26T01:08:59.800 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.800 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.800 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.800 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.800 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.800 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.800 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.059123+0000", 2024-04-26T01:08:59.800 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.800 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.800 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.800 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.059123+0000", 2024-04-26T01:08:59.800 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.801 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.801 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.801 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.059124+0000", 2024-04-26T01:08:59.801 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0839999999999999e-06 2024-04-26T01:08:59.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.801 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.801 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.801 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.059964+0000", 2024-04-26T01:08:59.801 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00083960600000000001 2024-04-26T01:08:59.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.801 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.801 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.801 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066110+0000", 2024-04-26T01:08:59.802 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0061464900000000001 2024-04-26T01:08:59.802 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.802 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.802 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066137+0000", 2024-04-26T01:08:59.802 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6307000000000001e-05 2024-04-26T01:08:59.802 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.802 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.802 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066336+0000", 2024-04-26T01:08:59.802 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019909100000000001 2024-04-26T01:08:59.802 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.802 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.803 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066365+0000", 2024-04-26T01:08:59.803 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9490000000000001e-05 2024-04-26T01:08:59.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.803 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.803 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.803 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.073999+0000", 2024-04-26T01:08:59.803 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0076337089999999998 2024-04-26T01:08:59.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.803 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.803 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.803 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.074104+0000", 2024-04-26T01:08:59.803 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000104607 2024-04-26T01:08:59.803 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.804 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.804 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.804 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.804 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.804 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 50/44 ECSubWrite(tid=219, reqid=osd.4.0:219, at_version=50'58, trim_to=0'0, roll_forward_to=47'56))", 2024-04-26T01:08:59.804 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.065984+0000", 2024-04-26T01:08:59.804 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 43.683041183, 2024-04-26T01:08:59.804 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.018105768000000001, 2024-04-26T01:08:59.804 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.804 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.804 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.804 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.804 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.804 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.804 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065984+0000", 2024-04-26T01:08:59.805 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.805 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.805 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.805 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.805 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065984+0000", 2024-04-26T01:08:59.805 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.805 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.805 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.805 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.805 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.065984+0000", 2024-04-26T01:08:59.805 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.7700000000000004e-07 2024-04-26T01:08:59.805 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.805 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.806 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.806 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066035+0000", 2024-04-26T01:08:59.806 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.0470000000000003e-05 2024-04-26T01:08:59.806 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.806 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.806 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.806 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077389+0000", 2024-04-26T01:08:59.806 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011354326 2024-04-26T01:08:59.806 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.806 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.806 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.807 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077419+0000", 2024-04-26T01:08:59.807 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.0006e-05 2024-04-26T01:08:59.807 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.807 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.807 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.807 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077629+0000", 2024-04-26T01:08:59.807 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000209471 2024-04-26T01:08:59.807 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.807 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.807 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.807 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077685+0000", 2024-04-26T01:08:59.807 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.6674e-05 2024-04-26T01:08:59.807 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.807 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.807 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.808 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.083973+0000", 2024-04-26T01:08:59.808 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0062876870000000001 2024-04-26T01:08:59.808 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.808 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.808 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.808 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.084089+0000", 2024-04-26T01:08:59.808 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000116357 2024-04-26T01:08:59.808 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.808 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.808 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.808 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.808 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.808 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/30 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=50'32, trim_to=0'0, roll_forward_to=45'30))", 2024-04-26T01:08:59.808 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.066181+0000", 2024-04-26T01:08:59.809 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 43.682843761999997, 2024-04-26T01:08:59.809 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011395499, 2024-04-26T01:08:59.809 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.809 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.809 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.809 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.809 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.809 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.809 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066181+0000", 2024-04-26T01:08:59.809 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.809 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.810 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.810 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.810 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066181+0000", 2024-04-26T01:08:59.810 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.810 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.810 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.810 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066183+0000", 2024-04-26T01:08:59.810 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.767e-06 2024-04-26T01:08:59.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.810 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.811 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.811 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066240+0000", 2024-04-26T01:08:59.811 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.7099999999999999e-05 2024-04-26T01:08:59.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.811 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.811 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.811 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066338+0000", 2024-04-26T01:08:59.811 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7738000000000005e-05 2024-04-26T01:08:59.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.811 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.811 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.811 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.066361+0000", 2024-04-26T01:08:59.811 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3615999999999999e-05 2024-04-26T01:08:59.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.812 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.812 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.812 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.070266+0000", 2024-04-26T01:08:59.812 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0039045659999999999 2024-04-26T01:08:59.812 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.812 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.812 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.812 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.070819+0000", 2024-04-26T01:08:59.812 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00055286599999999995 2024-04-26T01:08:59.812 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.812 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.812 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.812 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077394+0000", 2024-04-26T01:08:59.812 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0065747990000000001 2024-04-26T01:08:59.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.813 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.813 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.813 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.077577+0000", 2024-04-26T01:08:59.813 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00018304700000000001 2024-04-26T01:08:59.813 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.813 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.813 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.813 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.813 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 70/44 ECSubWrite(tid=411, reqid=client.4521.0:727, at_version=0'0, trim_to=0'0, roll_forward_to=70'88))", 2024-04-26T01:08:59.813 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:36.073457+0000", 2024-04-26T01:08:59.813 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 23.675567686000001, 2024-04-26T01:08:59.813 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.12471196599999999, 2024-04-26T01:08:59.813 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.814 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.814 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.814 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.814 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.814 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.814 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:36.073457+0000", 2024-04-26T01:08:59.814 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.814 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.814 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.814 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.814 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:36.073457+0000", 2024-04-26T01:08:59.814 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.814 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.815 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.815 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.815 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:36.073463+0000", 2024-04-26T01:08:59.815 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.5940000000000001e-06 2024-04-26T01:08:59.815 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.815 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.815 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.815 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:36.073480+0000", 2024-04-26T01:08:59.815 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6966e-05 2024-04-26T01:08:59.815 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.815 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.815 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.815 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:36.197284+0000", 2024-04-26T01:08:59.815 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.123803923 2024-04-26T01:08:59.815 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.816 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.816 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.816 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:36.197307+0000", 2024-04-26T01:08:59.816 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2997e-05 2024-04-26T01:08:59.816 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.816 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.816 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.816 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:36.197391+0000", 2024-04-26T01:08:59.816 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.4421999999999999e-05 2024-04-26T01:08:59.816 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.816 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.817 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.817 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:36.197400+0000", 2024-04-26T01:08:59.817 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.6340000000000007e-06 2024-04-26T01:08:59.817 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.817 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.817 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.817 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:36.198123+0000", 2024-04-26T01:08:59.817 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000723688 2024-04-26T01:08:59.817 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.817 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.817 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.817 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:36.198169+0000", 2024-04-26T01:08:59.818 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.5741999999999997e-05 2024-04-26T01:08:59.818 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.818 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.818 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.818 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.818 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.818 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:778 3.8s0 3:11b43273:::smithi16428812-17:head [read 0~81920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:08:59.818 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:38.243651+0000", 2024-04-26T01:08:59.818 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.505373893000002, 2024-04-26T01:08:59.818 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011106894000000001, 2024-04-26T01:08:59.818 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.818 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.818 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:59.819 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:59.819 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:59.819 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:59.819 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 778 2024-04-26T01:08:59.819 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.819 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.820 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.820 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.820 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.243651+0000", 2024-04-26T01:08:59.820 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.820 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.820 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.820 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.820 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.243651+0000", 2024-04-26T01:08:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.243652+0000", 2024-04-26T01:08:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1969999999999999e-06 2024-04-26T01:08:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.243661+0000", 2024-04-26T01:08:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2699999999999993e-06 2024-04-26T01:08:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.243678+0000", 2024-04-26T01:08:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.694e-05 2024-04-26T01:08:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.243688+0000", 2024-04-26T01:08:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.9899999999999992e-06 2024-04-26T01:08:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.243965+0000", 2024-04-26T01:08:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00027641599999999998 2024-04-26T01:08:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.243985+0000", 2024-04-26T01:08:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9953999999999999e-05 2024-04-26T01:08:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.251076+0000", 2024-04-26T01:08:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0070917130000000004 2024-04-26T01:08:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.251297+0000", 2024-04-26T01:08:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000220595 2024-04-26T01:08:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.254758+0000", 2024-04-26T01:08:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.003460819 2024-04-26T01:08:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:779 3.8s0 3:11b43273:::smithi16428812-17:head [read 0~81920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:08:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:38.243706+0000", 2024-04-26T01:08:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.505318984999999, 2024-04-26T01:08:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.014469691, 2024-04-26T01:08:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:08:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 779 2024-04-26T01:08:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.243706+0000", 2024-04-26T01:08:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.243706+0000", 2024-04-26T01:08:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.243707+0000", 2024-04-26T01:08:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.9699999999999995e-07 2024-04-26T01:08:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.243711+0000", 2024-04-26T01:08:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3499999999999999e-06 2024-04-26T01:08:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.243721+0000", 2024-04-26T01:08:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.8530000000000004e-06 2024-04-26T01:08:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.243729+0000", 2024-04-26T01:08:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.2230000000000007e-06 2024-04-26T01:08:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.244044+0000", 2024-04-26T01:08:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00031541300000000001 2024-04-26T01:08:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:08:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.244054+0000", 2024-04-26T01:08:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.8840000000000006e-06 2024-04-26T01:08:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.254891+0000", 2024-04-26T01:08:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010836691000000001 2024-04-26T01:08:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.255076+0000", 2024-04-26T01:08:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000185331 2024-04-26T01:08:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.258176+0000", 2024-04-26T01:08:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0030991489999999998 2024-04-26T01:08:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:782 3.8s0 3:1b6b0095:::smithi16428812-3:head [copy-from ver 4] snapc 15=[15] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:08:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:38.254430+0000", 2024-04-26T01:08:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 21.494595126, 2024-04-26T01:08:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011319279, 2024-04-26T01:08:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 782 2024-04-26T01:08:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.254430+0000", 2024-04-26T01:08:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.254430+0000", 2024-04-26T01:08:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.254432+0000", 2024-04-26T01:08:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5900000000000002e-06 2024-04-26T01:08:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.254446+0000", 2024-04-26T01:08:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3934000000000001e-05 2024-04-26T01:08:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.254480+0000", 2024-04-26T01:08:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.4066999999999998e-05 2024-04-26T01:08:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.254505+0000", 2024-04-26T01:08:59.834 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4627e-05 2024-04-26T01:08:59.834 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.834 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.834 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.834 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.255700+0000", 2024-04-26T01:08:59.834 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0011948309999999999 2024-04-26T01:08:59.834 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.834 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.834 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:59.834 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.255902+0000", 2024-04-26T01:08:59.834 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00020211799999999999 2024-04-26T01:08:59.834 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.834 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.835 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.835 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.260701+0000", 2024-04-26T01:08:59.835 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0047993180000000003 2024-04-26T01:08:59.835 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.835 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.835 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.835 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.264601+0000", 2024-04-26T01:08:59.835 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0038998969999999998 2024-04-26T01:08:59.835 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.835 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.835 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:59.835 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.265541+0000", 2024-04-26T01:08:59.835 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00093971799999999998 2024-04-26T01:08:59.835 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.835 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.836 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.836 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:38.265749+0000", 2024-04-26T01:08:59.836 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00020817899999999999 2024-04-26T01:08:59.836 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.836 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.836 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.836 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.836 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.836 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.0s4 81/30 ECSubWrite(tid=103, reqid=osd.14.0:103, at_version=81'80, trim_to=0'0, roll_forward_to=80'78))", 2024-04-26T01:08:59.836 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:47.312783+0000", 2024-04-26T01:08:59.836 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 12.436242019, 2024-04-26T01:08:59.836 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.026705908, 2024-04-26T01:08:59.836 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.837 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.837 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.837 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.837 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.837 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.837 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.312783+0000", 2024-04-26T01:08:59.837 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.837 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.837 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.837 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.838 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.312783+0000", 2024-04-26T01:08:59.838 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.838 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.838 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.838 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.838 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.312783+0000", 2024-04-26T01:08:59.838 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.0699999999999997e-07 2024-04-26T01:08:59.838 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.838 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.838 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.838 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.312838+0000", 2024-04-26T01:08:59.838 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.4384e-05 2024-04-26T01:08:59.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.839 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.839 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.839 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.312952+0000", 2024-04-26T01:08:59.839 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00011401 2024-04-26T01:08:59.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.839 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.839 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.839 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.312977+0000", 2024-04-26T01:08:59.840 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5531e-05 2024-04-26T01:08:59.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.840 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.840 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.840 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.318622+0000", 2024-04-26T01:08:59.840 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0056446730000000002 2024-04-26T01:08:59.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.841 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.841 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.841 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.318648+0000", 2024-04-26T01:08:59.841 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.5517e-05 2024-04-26T01:08:59.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.841 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.841 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.841 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.339403+0000", 2024-04-26T01:08:59.841 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.020755262999999999 2024-04-26T01:08:59.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.841 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.841 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.842 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.339489+0000", 2024-04-26T01:08:59.842 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.5922999999999997e-05 2024-04-26T01:08:59.842 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.842 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.842 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.842 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.842 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.842 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:899 3.8s0 3:1b6b0095:::smithi16428812-3:head [append 0~0] snapc 1b=[1b,1a,19,18] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:08:59.842 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:47.482292+0000", 2024-04-26T01:08:59.842 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 12.26673289, 2024-04-26T01:08:59.842 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.015260226999999999, 2024-04-26T01:08:59.842 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.843 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.843 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:59.843 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:59.843 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:59.843 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:59.843 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 899 2024-04-26T01:08:59.843 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.843 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.843 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.843 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.843 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482292+0000", 2024-04-26T01:08:59.843 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.843 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.843 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.844 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.844 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482292+0000", 2024-04-26T01:08:59.844 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.844 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.844 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.844 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.844 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482297+0000", 2024-04-26T01:08:59.844 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.9030000000000003e-06 2024-04-26T01:08:59.844 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.844 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.844 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.844 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482312+0000", 2024-04-26T01:08:59.845 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5403000000000001e-05 2024-04-26T01:08:59.845 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.845 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.845 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.845 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482364+0000", 2024-04-26T01:08:59.845 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.1823999999999997e-05 2024-04-26T01:08:59.845 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.846 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.846 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.846 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482387+0000", 2024-04-26T01:08:59.846 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3067000000000001e-05 2024-04-26T01:08:59.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.847 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.847 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.847 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482943+0000", 2024-04-26T01:08:59.847 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00055636300000000004 2024-04-26T01:08:59.847 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.847 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.848 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:59.848 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.483219+0000", 2024-04-26T01:08:59.848 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00027512499999999998 2024-04-26T01:08:59.848 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.848 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.848 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.849 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.485078+0000", 2024-04-26T01:08:59.849 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001859143 2024-04-26T01:08:59.849 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.849 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.849 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.849 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.496985+0000", 2024-04-26T01:08:59.850 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011907386000000001 2024-04-26T01:08:59.850 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.850 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.850 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:59.850 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.497292+0000", 2024-04-26T01:08:59.851 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00030649600000000001 2024-04-26T01:08:59.851 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.851 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.851 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.851 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.497552+0000", 2024-04-26T01:08:59.851 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00026051699999999999 2024-04-26T01:08:59.852 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.852 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.852 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.852 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.852 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.852 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:900 3.8s0 3:1b6b0095:::smithi16428812-3:head [rollback 1a] snapc 1b=[1b,1a,19,18] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:08:59.853 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:47.482445+0000", 2024-04-26T01:08:59.853 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 12.266579663, 2024-04-26T01:08:59.853 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.018663715000000001, 2024-04-26T01:08:59.853 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.853 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.853 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:59.854 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:59.854 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:59.854 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:59.854 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 900 2024-04-26T01:08:59.854 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.854 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.855 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.855 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.855 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482445+0000", 2024-04-26T01:08:59.855 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.855 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.855 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.856 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.856 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482445+0000", 2024-04-26T01:08:59.856 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.856 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.856 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.856 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.857 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482447+0000", 2024-04-26T01:08:59.857 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1770000000000001e-06 2024-04-26T01:08:59.857 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.857 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.857 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.857 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482456+0000", 2024-04-26T01:08:59.858 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.9730000000000003e-06 2024-04-26T01:08:59.858 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.858 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.858 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.858 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482481+0000", 2024-04-26T01:08:59.858 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4749999999999999e-05 2024-04-26T01:08:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482498+0000", 2024-04-26T01:08:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7289999999999999e-05 2024-04-26T01:08:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.491944+0000", 2024-04-26T01:08:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0094459599999999998 2024-04-26T01:08:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.492233+0000", 2024-04-26T01:08:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00028912600000000001 2024-04-26T01:08:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.494009+0000", 2024-04-26T01:08:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0017751749999999999 2024-04-26T01:08:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.499950+0000", 2024-04-26T01:08:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0059411050000000003 2024-04-26T01:08:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.500887+0000", 2024-04-26T01:08:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00093761500000000004 2024-04-26T01:08:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.501109+0000", 2024-04-26T01:08:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022154400000000001 2024-04-26T01:08:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:901 3.8s0 3:1b6b0095:::smithi16428812-3:head [append 0~0] snapc 1b=[1b,1a,19,18] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:08:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:47.482527+0000", 2024-04-26T01:08:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 12.266497626, 2024-04-26T01:08:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.024230732000000001, 2024-04-26T01:08:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:08:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:08:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:08:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:08:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 901 2024-04-26T01:08:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482527+0000", 2024-04-26T01:08:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482527+0000", 2024-04-26T01:08:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482529+0000", 2024-04-26T01:08:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.367e-06 2024-04-26T01:08:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482539+0000", 2024-04-26T01:08:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0423e-05 2024-04-26T01:08:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482561+0000", 2024-04-26T01:08:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1929999999999998e-05 2024-04-26T01:08:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.482584+0000", 2024-04-26T01:08:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3093999999999999e-05 2024-04-26T01:08:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.498023+0000", 2024-04-26T01:08:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.015439158 2024-04-26T01:08:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:08:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.498252+0000", 2024-04-26T01:08:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022853500000000001 2024-04-26T01:08:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.499497+0000", 2024-04-26T01:08:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001244956 2024-04-26T01:08:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.504873+0000", 2024-04-26T01:08:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0053765319999999998 2024-04-26T01:08:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:08:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.505072+0000", 2024-04-26T01:08:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019863500000000001 2024-04-26T01:08:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.506758+0000", 2024-04-26T01:08:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0016861020000000001 2024-04-26T01:08:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 81/44 ECSubWrite(tid=477, reqid=client.4521.0:895, at_version=81'99, trim_to=0'0, roll_forward_to=81'97))", 2024-04-26T01:08:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:47.484836+0000", 2024-04-26T01:08:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 12.264188692999999, 2024-04-26T01:08:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011014548000000001, 2024-04-26T01:08:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.484836+0000", 2024-04-26T01:08:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.484836+0000", 2024-04-26T01:08:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.484838+0000", 2024-04-26T01:08:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1900000000000002e-06 2024-04-26T01:08:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.485300+0000", 2024-04-26T01:08:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00046203500000000002 2024-04-26T01:08:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.485389+0000", 2024-04-26T01:08:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.8704999999999999e-05 2024-04-26T01:08:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.485413+0000", 2024-04-26T01:08:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3513e-05 2024-04-26T01:08:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.489042+0000", 2024-04-26T01:08:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0036298390000000002 2024-04-26T01:08:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.489071+0000", 2024-04-26T01:08:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8427000000000001e-05 2024-04-26T01:08:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.495714+0000", 2024-04-26T01:08:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0066436309999999997 2024-04-26T01:08:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.495851+0000", 2024-04-26T01:08:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013620799999999999 2024-04-26T01:08:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.1s3 81/44 ECSubWrite(tid=479, reqid=client.4521.0:904, at_version=81'101, trim_to=0'0, roll_forward_to=81'97))", 2024-04-26T01:08:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:47.486524+0000", 2024-04-26T01:08:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 12.262500450999999, 2024-04-26T01:08:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013451057000000001, 2024-04-26T01:08:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:08:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:08:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:08:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:08:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:08:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.486524+0000", 2024-04-26T01:08:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:08:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.486524+0000", 2024-04-26T01:08:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:08:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:08:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.486527+0000", 2024-04-26T01:08:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.204e-06 2024-04-26T01:08:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:08:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.486578+0000", 2024-04-26T01:08:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.1536000000000003e-05 2024-04-26T01:08:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:08:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.489145+0000", 2024-04-26T01:08:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0025665309999999999 2024-04-26T01:08:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:08:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.489167+0000", 2024-04-26T01:08:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2337e-05 2024-04-26T01:08:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:08:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.491040+0000", 2024-04-26T01:08:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0018730260000000001 2024-04-26T01:08:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:08:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.491065+0000", 2024-04-26T01:08:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.508e-05 2024-04-26T01:08:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:08:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.499891+0000", 2024-04-26T01:08:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0088257500000000003 2024-04-26T01:08:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:08:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:08:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:08:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.499975+0000", 2024-04-26T01:08:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.4592999999999995e-05 2024-04-26T01:08:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:08:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:08:59.876 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:08:59.877 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:08:59.878 INFO:tasks.ceph.osd.8.smithi008.stderr:2024-04-26T01:08:59.872+0000 7f1c2c3b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25002) UID: 0 2024-04-26T01:08:59.978 INFO:tasks.ceph.osd.12.smithi008.stderr:2024-04-26T01:08:59.976+0000 7f2dbcaf3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25004) UID: 0 2024-04-26T01:09:00.011 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:09:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:09:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: "num_ops": 0 2024-04-26T01:09:00.011 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:09:00.026 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:09:00.079 INFO:tasks.ceph.osd.4.smithi008.stderr:2024-04-26T01:09:00.076+0000 7f6ef4c9c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25000) UID: 0 2024-04-26T01:09:00.080 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'unique_pool_0' removed 2024-04-26T01:09:00.080 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-04-26T01:09:00.117 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-26T01:09:00.126 INFO:tasks.thrashosds:joining thrashosds 2024-04-26T01:09:00.151 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:09:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:09:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:09:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:09:00.152 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:09:00.165 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:09:00.294 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:09:00.294 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:09:00.294 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:09:00.294 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:09:00.294 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.294 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/30 ECSubWrite(tid=9, reqid=client.4521.0:38, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:09:00.294 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.327957+0000", 2024-04-26T01:09:00.294 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 60.964466991000002, 2024-04-26T01:09:00.294 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.029348438000000001, 2024-04-26T01:09:00.294 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.294 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.295 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:09:00.295 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.295 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.295 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.295 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327957+0000", 2024-04-26T01:09:00.295 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.295 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.295 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.295 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327957+0000", 2024-04-26T01:09:00.295 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.295 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.295 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.296 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327958+0000", 2024-04-26T01:09:00.296 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.6300000000000005e-07 2024-04-26T01:09:00.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.296 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.296 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.296 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.327988+0000", 2024-04-26T01:09:00.296 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.0577000000000002e-05 2024-04-26T01:09:00.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.296 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.296 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.296 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.354381+0000", 2024-04-26T01:09:00.296 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.026392280000000001 2024-04-26T01:09:00.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.297 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.297 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.297 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.354407+0000", 2024-04-26T01:09:00.297 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6783000000000001e-05 2024-04-26T01:09:00.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.297 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.297 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.297 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.354542+0000", 2024-04-26T01:09:00.297 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013464299999999999 2024-04-26T01:09:00.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.297 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.297 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:09:00.297 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.354563+0000", 2024-04-26T01:09:00.297 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0744999999999999e-05 2024-04-26T01:09:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:09:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357241+0000", 2024-04-26T01:09:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0026777929999999999 2024-04-26T01:09:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.357306+0000", 2024-04-26T01:09:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.5054000000000004e-05 2024-04-26T01:09:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpRead(3.1s2 33/30 ECSubRead(tid=30, to_read={3:88871701:::smithi16428812-34:head=0,4096,0}, subchunks={3:88871701:::smithi16428812-34:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:09:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.420990+0000", 2024-04-26T01:09:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 60.871434336, 2024-04-26T01:09:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.028504367999999999, 2024-04-26T01:09:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:09:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.420990+0000", 2024-04-26T01:09:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.300 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.420990+0000", 2024-04-26T01:09:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.420991+0000", 2024-04-26T01:09:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.254e-06 2024-04-26T01:09:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.421000+0000", 2024-04-26T01:09:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.8370000000000003e-06 2024-04-26T01:09:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.448556+0000", 2024-04-26T01:09:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.027556243000000001 2024-04-26T01:09:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.448583+0000", 2024-04-26T01:09:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6761e-05 2024-04-26T01:09:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.449059+0000", 2024-04-26T01:09:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00047558499999999999 2024-04-26T01:09:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.449494+0000", 2024-04-26T01:09:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00043568799999999999 2024-04-26T01:09:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:487 3.cs0 3:3bb987cc:::smithi16428812-47:head [read 0~1884160,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-04-26T01:09:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.037904+0000", 2024-04-26T01:09:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 44.254520255999999, 2024-04-26T01:09:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.044199702, 2024-04-26T01:09:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:09:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 487 2024-04-26T01:09:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037904+0000", 2024-04-26T01:09:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037904+0000", 2024-04-26T01:09:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037908+0000", 2024-04-26T01:09:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.1829999999999996e-06 2024-04-26T01:09:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037926+0000", 2024-04-26T01:09:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7859999999999998e-05 2024-04-26T01:09:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037974+0000", 2024-04-26T01:09:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.8494e-05 2024-04-26T01:09:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.037999+0000", 2024-04-26T01:09:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4899999999999999e-05 2024-04-26T01:09:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038165+0000", 2024-04-26T01:09:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016607699999999999 2024-04-26T01:09:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op must wait for map", 2024-04-26T01:09:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038179+0000", 2024-04-26T01:09:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.403e-05 2024-04-26T01:09:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.049988+0000", 2024-04-26T01:09:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.011808206999999999 2024-04-26T01:09:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.050141+0000", 2024-04-26T01:09:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015328799999999999 2024-04-26T01:09:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.082104+0000", 2024-04-26T01:09:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.031962663000000002 2024-04-26T01:09:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:488 3.cs0 3:3bb987cc:::smithi16428812-47:head [read 0~1884160,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-04-26T01:09:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.038063+0000", 2024-04-26T01:09:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 44.254361692000003, 2024-04-26T01:09:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.047336889, 2024-04-26T01:09:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:09:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 488 2024-04-26T01:09:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038063+0000", 2024-04-26T01:09:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038063+0000", 2024-04-26T01:09:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.312 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.312 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038064+0000", 2024-04-26T01:09:00.312 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3799999999999999e-06 2024-04-26T01:09:00.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.312 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.312 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.312 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038075+0000", 2024-04-26T01:09:00.312 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0993e-05 2024-04-26T01:09:00.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.312 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.312 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.312 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038096+0000", 2024-04-26T01:09:00.312 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1284000000000002e-05 2024-04-26T01:09:00.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.313 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.313 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.313 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038112+0000", 2024-04-26T01:09:00.313 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5577e-05 2024-04-26T01:09:00.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.313 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.313 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.313 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038366+0000", 2024-04-26T01:09:00.313 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025445099999999999 2024-04-26T01:09:00.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.313 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.313 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:09:00.313 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038390+0000", 2024-04-26T01:09:00.313 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3782999999999999e-05 2024-04-26T01:09:00.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.314 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.314 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.314 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051105+0000", 2024-04-26T01:09:00.314 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012714757 2024-04-26T01:09:00.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.314 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.314 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.314 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051362+0000", 2024-04-26T01:09:00.314 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00025746800000000001 2024-04-26T01:09:00.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.314 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.314 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.314 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.085399+0000", 2024-04-26T01:09:00.315 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.034037195999999999 2024-04-26T01:09:00.315 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.315 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.315 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.315 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.315 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.315 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:489 3.cs0 3:3bb987cc:::smithi16428812-47:head [read 0~1884160,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-04-26T01:09:00.315 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.038135+0000", 2024-04-26T01:09:00.315 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 44.254289581999998, 2024-04-26T01:09:00.315 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.049787337000000001, 2024-04-26T01:09:00.315 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.315 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.315 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:09:00.315 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.315 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 489 2024-04-26T01:09:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038135+0000", 2024-04-26T01:09:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038135+0000", 2024-04-26T01:09:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038136+0000", 2024-04-26T01:09:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1739999999999999e-06 2024-04-26T01:09:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038144+0000", 2024-04-26T01:09:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.746e-06 2024-04-26T01:09:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038167+0000", 2024-04-26T01:09:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2977e-05 2024-04-26T01:09:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038182+0000", 2024-04-26T01:09:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5543e-05 2024-04-26T01:09:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038497+0000", 2024-04-26T01:09:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00031515200000000002 2024-04-26T01:09:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:09:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.038521+0000", 2024-04-26T01:09:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3706999999999999e-05 2024-04-26T01:09:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052089+0000", 2024-04-26T01:09:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013568347 2024-04-26T01:09:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.052357+0000", 2024-04-26T01:09:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00026783200000000002 2024-04-26T01:09:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.087922+0000", 2024-04-26T01:09:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.035564858999999997 2024-04-26T01:09:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpRead(3.cs0 50/44 ECSubRead(tid=19, to_read={3:3bb987cc:::smithi16428812-47:head=0,471040,0}, subchunks={3:3bb987cc:::smithi16428812-47:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:09:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.050864+0000", 2024-04-26T01:09:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 44.241560573999998, 2024-04-26T01:09:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.023549771000000001, 2024-04-26T01:09:00.324 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.324 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.324 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:09:00.324 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.324 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.324 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.324 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.050864+0000", 2024-04-26T01:09:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.050864+0000", 2024-04-26T01:09:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.050864+0000", 2024-04-26T01:09:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.050864+0000", 2024-04-26T01:09:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053143+0000", 2024-04-26T01:09:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0022794400000000002 2024-04-26T01:09:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053173+0000", 2024-04-26T01:09:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.9635999999999999e-05 2024-04-26T01:09:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.069570+0000", 2024-04-26T01:09:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.016397047000000001 2024-04-26T01:09:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.074413+0000", 2024-04-26T01:09:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0048436479999999999 2024-04-26T01:09:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpRead(3.cs0 50/44 ECSubRead(tid=20, to_read={3:3bb987cc:::smithi16428812-47:head=0,471040,0}, subchunks={3:3bb987cc:::smithi16428812-47:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:09:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.051911+0000", 2024-04-26T01:09:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 44.240512725999999, 2024-04-26T01:09:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.023435303000000001, 2024-04-26T01:09:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:09:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051911+0000", 2024-04-26T01:09:00.330 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.330 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.330 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.330 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051911+0000", 2024-04-26T01:09:00.330 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.330 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.330 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.330 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051911+0000", 2024-04-26T01:09:00.330 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.330 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.330 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.331 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.051911+0000", 2024-04-26T01:09:00.331 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.331 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.331 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053257+0000", 2024-04-26T01:09:00.331 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001345759 2024-04-26T01:09:00.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.331 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.331 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.053301+0000", 2024-04-26T01:09:00.331 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.3482999999999997e-05 2024-04-26T01:09:00.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.332 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.332 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075035+0000", 2024-04-26T01:09:00.332 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.021734690000000001 2024-04-26T01:09:00.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.332 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.332 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.332 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.075347+0000", 2024-04-26T01:09:00.332 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00031137100000000001 2024-04-26T01:09:00.332 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.332 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.332 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.332 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.332 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:09:00.333 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.055120+0000", 2024-04-26T01:09:00.333 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 44.237304645999998, 2024-04-26T01:09:00.333 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.023938912999999999, 2024-04-26T01:09:00.333 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.333 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.333 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:09:00.333 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.333 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.333 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.333 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055120+0000", 2024-04-26T01:09:00.333 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.333 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.333 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.333 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055120+0000", 2024-04-26T01:09:00.334 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.334 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.334 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055121+0000", 2024-04-26T01:09:00.334 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.367e-06 2024-04-26T01:09:00.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.334 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.334 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055787+0000", 2024-04-26T01:09:00.334 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000665702 2024-04-26T01:09:00.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.335 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.335 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063149+0000", 2024-04-26T01:09:00.335 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0073619590000000004 2024-04-26T01:09:00.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.335 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.335 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063177+0000", 2024-04-26T01:09:00.335 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8592999999999999e-05 2024-04-26T01:09:00.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.335 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.335 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.078938+0000", 2024-04-26T01:09:00.335 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.015760687999999998 2024-04-26T01:09:00.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.336 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.336 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.079058+0000", 2024-04-26T01:09:00.336 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000120604 2024-04-26T01:09:00.336 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.336 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.336 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.336 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.336 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 50/44 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:09:00.336 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.055795+0000", 2024-04-26T01:09:00.336 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 44.236629049999998, 2024-04-26T01:09:00.336 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.02721587, 2024-04-26T01:09:00.336 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.336 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.337 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:09:00.337 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.337 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.337 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055795+0000", 2024-04-26T01:09:00.337 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.337 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.337 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055795+0000", 2024-04-26T01:09:00.337 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.337 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.337 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.055796+0000", 2024-04-26T01:09:00.338 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.057e-06 2024-04-26T01:09:00.338 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.338 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.338 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.338 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.056601+0000", 2024-04-26T01:09:00.338 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00080521399999999995 2024-04-26T01:09:00.338 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.338 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.338 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.338 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063232+0000", 2024-04-26T01:09:00.338 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0066301579999999997 2024-04-26T01:09:00.338 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.338 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.339 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.339 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.063246+0000", 2024-04-26T01:09:00.339 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4124000000000001e-05 2024-04-26T01:09:00.339 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.339 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.339 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.339 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.082918+0000", 2024-04-26T01:09:00.339 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.019672486999999999 2024-04-26T01:09:00.339 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.339 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.339 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.339 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.083011+0000", 2024-04-26T01:09:00.339 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.2830000000000004e-05 2024-04-26T01:09:00.339 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.339 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.340 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.340 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.340 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.7s2 58/44 ECSubWrite(tid=309, reqid=osd.4.0:309, at_version=58'58, trim_to=0'0, roll_forward_to=58'54))", 2024-04-26T01:09:00.340 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:24.213069+0000", 2024-04-26T01:09:00.340 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 36.079354866999999, 2024-04-26T01:09:00.340 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.14087053199999999, 2024-04-26T01:09:00.340 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.340 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.340 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:09:00.340 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.340 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.340 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.341 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.213069+0000", 2024-04-26T01:09:00.341 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.341 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.341 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.341 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.213069+0000", 2024-04-26T01:09:00.341 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.341 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.341 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.341 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.213070+0000", 2024-04-26T01:09:00.341 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.44e-07 2024-04-26T01:09:00.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.341 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.341 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.342 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.213092+0000", 2024-04-26T01:09:00.342 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1849999999999999e-05 2024-04-26T01:09:00.342 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.342 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.342 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.342 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.350821+0000", 2024-04-26T01:09:00.342 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.13772947099999999 2024-04-26T01:09:00.342 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.342 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.342 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.342 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.350840+0000", 2024-04-26T01:09:00.342 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.927e-05 2024-04-26T01:09:00.343 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.343 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.343 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.343 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.351016+0000", 2024-04-26T01:09:00.343 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017531400000000001 2024-04-26T01:09:00.343 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.343 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.343 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:09:00.343 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.351047+0000", 2024-04-26T01:09:00.343 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.1099999999999997e-05 2024-04-26T01:09:00.343 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.343 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.343 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:09:00.343 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.353853+0000", 2024-04-26T01:09:00.344 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0028058359999999999 2024-04-26T01:09:00.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.344 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.344 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.344 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.353940+0000", 2024-04-26T01:09:00.344 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.7146999999999998e-05 2024-04-26T01:09:00.344 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.344 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.344 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.344 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.344 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.bs6 58/30 ECSubWrite(tid=318, reqid=client.4521.0:601, at_version=58'41, trim_to=0'0, roll_forward_to=58'39))", 2024-04-26T01:09:00.344 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:24.361342+0000", 2024-04-26T01:09:00.344 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 35.931082521999997, 2024-04-26T01:09:00.344 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.76301632799999997, 2024-04-26T01:09:00.345 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.345 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.345 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:09:00.345 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.345 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.345 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.345 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.361342+0000", 2024-04-26T01:09:00.345 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.345 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.345 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.345 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.345 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.361342+0000", 2024-04-26T01:09:00.346 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.346 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.346 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.346 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.346 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.361343+0000", 2024-04-26T01:09:00.346 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.683e-06 2024-04-26T01:09:00.346 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.346 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.346 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.346 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:24.361642+0000", 2024-04-26T01:09:00.346 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00029905399999999999 2024-04-26T01:09:00.346 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.347 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.347 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.347 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.121642+0000", 2024-04-26T01:09:00.347 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.75999975099999995 2024-04-26T01:09:00.347 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.347 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.347 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.347 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.121662+0000", 2024-04-26T01:09:00.347 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9956e-05 2024-04-26T01:09:00.347 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.347 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.347 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.348 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.121819+0000", 2024-04-26T01:09:00.348 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015640700000000001 2024-04-26T01:09:00.348 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.348 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.348 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:09:00.348 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.121838+0000", 2024-04-26T01:09:00.348 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9697000000000001e-05 2024-04-26T01:09:00.348 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.348 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.348 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:09:00.348 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.124294+0000", 2024-04-26T01:09:00.348 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0024562329999999999 2024-04-26T01:09:00.348 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.348 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.124358+0000", 2024-04-26T01:09:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.3547000000000007e-05 2024-04-26T01:09:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:635 3.cs0 3:3ca9ce7a:::smithi16428812-23:head [setxattr _header (57) in=7b] snapc f=[f,d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-26T01:09:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:25.511495+0000", 2024-04-26T01:09:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 34.780928733000003, 2024-04-26T01:09:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.26064569300000001, 2024-04-26T01:09:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:00.350 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.350 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.350 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.350 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 635 2024-04-26T01:09:00.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.350 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.350 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.350 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.350 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511495+0000", 2024-04-26T01:09:00.350 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.350 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.350 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.351 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511495+0000", 2024-04-26T01:09:00.351 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.351 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.351 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.351 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.351 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511497+0000", 2024-04-26T01:09:00.351 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2130000000000001e-06 2024-04-26T01:09:00.351 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.351 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.351 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.351 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511502+0000", 2024-04-26T01:09:00.351 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.4999999999999999e-06 2024-04-26T01:09:00.351 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.351 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.352 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.352 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511516+0000", 2024-04-26T01:09:00.352 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.402e-05 2024-04-26T01:09:00.352 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.352 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.352 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.352 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511525+0000", 2024-04-26T01:09:00.352 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.0869999999999996e-06 2024-04-26T01:09:00.352 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.352 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.352 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.352 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.524239+0000", 2024-04-26T01:09:00.352 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.012713296000000001 2024-04-26T01:09:00.352 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.352 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.353 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.353 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.524539+0000", 2024-04-26T01:09:00.353 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000300501 2024-04-26T01:09:00.353 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.353 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.353 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:09:00.353 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.525917+0000", 2024-04-26T01:09:00.353 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0013776839999999999 2024-04-26T01:09:00.353 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.353 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.353 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:09:00.354 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.527728+0000", 2024-04-26T01:09:00.354 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0018110229999999999 2024-04-26T01:09:00.354 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.354 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.354 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:09:00.354 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.771988+0000", 2024-04-26T01:09:00.354 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.244260692 2024-04-26T01:09:00.354 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.354 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.354 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.354 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.772141+0000", 2024-04-26T01:09:00.354 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015267699999999999 2024-04-26T01:09:00.354 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.355 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.355 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.355 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.355 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.355 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:636 3.cs0 3:3ca9ce7a:::smithi16428812-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-04-26T01:09:00.355 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:25.511538+0000", 2024-04-26T01:09:00.355 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 34.780886563000003, 2024-04-26T01:09:00.355 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.26493635500000001, 2024-04-26T01:09:00.355 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.355 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.355 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:09:00.355 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.355 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.355 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.356 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 636 2024-04-26T01:09:00.356 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.356 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.356 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.356 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.356 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511538+0000", 2024-04-26T01:09:00.356 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.356 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.356 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.356 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.356 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511538+0000", 2024-04-26T01:09:00.356 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.356 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.356 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.357 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.357 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511538+0000", 2024-04-26T01:09:00.357 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8000000000000005e-07 2024-04-26T01:09:00.357 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.357 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.357 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.357 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511542+0000", 2024-04-26T01:09:00.357 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.0260000000000001e-06 2024-04-26T01:09:00.357 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.357 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.357 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.357 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511556+0000", 2024-04-26T01:09:00.357 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3784000000000001e-05 2024-04-26T01:09:00.357 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.357 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.358 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.358 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.511576+0000", 2024-04-26T01:09:00.358 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.948e-05 2024-04-26T01:09:00.358 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.358 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.358 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.358 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.526765+0000", 2024-04-26T01:09:00.358 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.015189393000000001 2024-04-26T01:09:00.358 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.358 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.358 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:00.358 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.526913+0000", 2024-04-26T01:09:00.359 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014794299999999999 2024-04-26T01:09:00.359 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.359 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.359 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.359 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.772915+0000", 2024-04-26T01:09:00.359 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.24600209100000001 2024-04-26T01:09:00.359 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.359 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.359 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.359 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.773181+0000", 2024-04-26T01:09:00.359 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00026546399999999998 2024-04-26T01:09:00.359 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.360 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.360 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.360 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:25.776474+0000", 2024-04-26T01:09:00.360 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0032933939999999998 2024-04-26T01:09:00.360 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.360 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.360 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.360 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.360 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.360 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:730 3.cs0 3:34b45d9f:::smithi16428812-33:head [append 0~622592] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:09:00.360 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:37.113296+0000", 2024-04-26T01:09:00.360 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 23.179128188, 2024-04-26T01:09:00.361 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.024412475999999999, 2024-04-26T01:09:00.361 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.361 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.361 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:00.361 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.361 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.361 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.361 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 730 2024-04-26T01:09:00.361 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.361 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.361 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.361 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.362 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.113296+0000", 2024-04-26T01:09:00.362 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.362 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.362 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.362 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.362 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.113296+0000", 2024-04-26T01:09:00.362 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.362 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.362 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.362 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.362 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.113298+0000", 2024-04-26T01:09:00.363 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9149999999999999e-06 2024-04-26T01:09:00.363 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.363 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.363 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.363 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.113931+0000", 2024-04-26T01:09:00.363 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00063304699999999997 2024-04-26T01:09:00.363 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.363 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.363 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.363 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.113952+0000", 2024-04-26T01:09:00.363 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0686000000000002e-05 2024-04-26T01:09:00.363 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.363 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.363 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.364 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.113962+0000", 2024-04-26T01:09:00.364 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0828e-05 2024-04-26T01:09:00.364 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.364 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.364 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.364 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.114041+0000", 2024-04-26T01:09:00.364 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8435000000000004e-05 2024-04-26T01:09:00.364 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.364 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.364 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.364 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.114191+0000", 2024-04-26T01:09:00.364 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015031699999999999 2024-04-26T01:09:00.364 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.364 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.365 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:09:00.365 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.115423+0000", 2024-04-26T01:09:00.365 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001231413 2024-04-26T01:09:00.365 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.365 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.365 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:09:00.365 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.118205+0000", 2024-04-26T01:09:00.365 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0027828050000000002 2024-04-26T01:09:00.365 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.365 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.365 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:09:00.365 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.137572+0000", 2024-04-26T01:09:00.365 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.019366471999999999 2024-04-26T01:09:00.365 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.366 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.366 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.366 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.137708+0000", 2024-04-26T01:09:00.366 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000136558 2024-04-26T01:09:00.366 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.366 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.366 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.366 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.366 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.366 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:734 3.cs0 3:34b45d9f:::smithi16428812-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:09:00.366 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:37.126379+0000", 2024-04-26T01:09:00.366 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 23.166045055000001, 2024-04-26T01:09:00.366 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.045415540999999997, 2024-04-26T01:09:00.367 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.367 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.367 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:09:00.367 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.367 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.367 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.367 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 734 2024-04-26T01:09:00.367 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.367 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.367 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.367 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.367 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.126379+0000", 2024-04-26T01:09:00.367 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.367 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.367 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.368 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.368 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.126379+0000", 2024-04-26T01:09:00.368 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.368 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.368 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.368 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.126380+0000", 2024-04-26T01:09:00.368 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.4499999999999996e-07 2024-04-26T01:09:00.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.368 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.368 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.368 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.126383+0000", 2024-04-26T01:09:00.368 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.5149999999999998e-06 2024-04-26T01:09:00.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.369 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.369 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.369 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.126394+0000", 2024-04-26T01:09:00.369 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0087e-05 2024-04-26T01:09:00.369 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.369 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.369 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.369 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.126401+0000", 2024-04-26T01:09:00.369 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.6120000000000001e-06 2024-04-26T01:09:00.369 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.369 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.369 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.369 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.129935+0000", 2024-04-26T01:09:00.369 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0035335200000000001 2024-04-26T01:09:00.369 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.370 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.370 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:00.370 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.130058+0000", 2024-04-26T01:09:00.370 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012322599999999999 2024-04-26T01:09:00.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.370 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.370 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.370 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.143272+0000", 2024-04-26T01:09:00.370 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013213997 2024-04-26T01:09:00.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.370 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.371 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.371 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.143368+0000", 2024-04-26T01:09:00.371 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6526000000000004e-05 2024-04-26T01:09:00.371 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.371 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.371 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.371 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.171795+0000", 2024-04-26T01:09:00.371 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.028426313000000002 2024-04-26T01:09:00.371 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.371 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.371 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.371 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.371 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.371 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.cs0 71/44 ECSubReadReply(tid=79, attrs_read=0))", 2024-04-26T01:09:00.371 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:37.144037+0000", 2024-04-26T01:09:00.372 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 23.148387455999998, 2024-04-26T01:09:00.372 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.027776942999999998, 2024-04-26T01:09:00.372 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.372 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.372 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:09:00.372 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.372 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.372 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.372 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.144037+0000", 2024-04-26T01:09:00.372 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.373 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.373 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.373 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.144037+0000", 2024-04-26T01:09:00.373 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.373 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.373 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.373 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.373 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.144038+0000", 2024-04-26T01:09:00.373 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6899999999999996e-07 2024-04-26T01:09:00.373 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.373 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.373 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.373 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.144052+0000", 2024-04-26T01:09:00.373 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4049999999999999e-05 2024-04-26T01:09:00.374 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.374 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.374 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.171458+0000", 2024-04-26T01:09:00.374 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.027406125999999999 2024-04-26T01:09:00.374 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.374 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.374 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.171470+0000", 2024-04-26T01:09:00.374 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2407e-05 2024-04-26T01:09:00.374 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.374 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.375 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.171593+0000", 2024-04-26T01:09:00.375 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012221700000000001 2024-04-26T01:09:00.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.375 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.375 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.375 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.171814+0000", 2024-04-26T01:09:00.375 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00022117400000000001 2024-04-26T01:09:00.375 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.375 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.375 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.375 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.375 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:870 3.cs0 3:34b45d9f:::smithi16428812-33:head [setxattr _header (57) in=7b] snapc 18=[18,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-04-26T01:09:00.375 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:43.341776+0000", 2024-04-26T01:09:00.375 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 16.950648304000001, 2024-04-26T01:09:00.376 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.077791930999999995, 2024-04-26T01:09:00.376 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.376 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.376 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:00.376 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.376 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.376 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.376 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 870 2024-04-26T01:09:00.376 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.376 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.376 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.376 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.376 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.341776+0000", 2024-04-26T01:09:00.376 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.376 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.377 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.377 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.377 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.341776+0000", 2024-04-26T01:09:00.377 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.377 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.377 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.377 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.341777+0000", 2024-04-26T01:09:00.377 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2100000000000001e-06 2024-04-26T01:09:00.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.377 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.377 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.377 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.341783+0000", 2024-04-26T01:09:00.377 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.2199999999999997e-06 2024-04-26T01:09:00.378 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.378 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.378 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.378 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.341794+0000", 2024-04-26T01:09:00.378 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0917000000000001e-05 2024-04-26T01:09:00.378 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.378 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.378 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.378 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.341802+0000", 2024-04-26T01:09:00.378 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1340000000000004e-06 2024-04-26T01:09:00.378 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.378 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.378 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.378 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.345804+0000", 2024-04-26T01:09:00.379 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.004001185 2024-04-26T01:09:00.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.379 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.379 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.379 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.345962+0000", 2024-04-26T01:09:00.379 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015798799999999999 2024-04-26T01:09:00.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.379 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.379 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:09:00.379 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.346766+0000", 2024-04-26T01:09:00.379 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00080424200000000004 2024-04-26T01:09:00.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.379 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.379 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:09:00.379 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.349067+0000", 2024-04-26T01:09:00.380 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0023014670000000002 2024-04-26T01:09:00.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.380 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.380 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:09:00.380 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.419379+0000", 2024-04-26T01:09:00.380 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.070311266999999997 2024-04-26T01:09:00.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.380 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.380 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.380 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.419568+0000", 2024-04-26T01:09:00.380 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00018930100000000001 2024-04-26T01:09:00.380 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.381 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.381 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.381 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.381 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.381 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:871 3.cs0 3:34b45d9f:::smithi16428812-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-04-26T01:09:00.381 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:43.341851+0000", 2024-04-26T01:09:00.381 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 16.950572949000001, 2024-04-26T01:09:00.381 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.082841497, 2024-04-26T01:09:00.381 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.381 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.381 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:09:00.381 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.381 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.381 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.381 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 871 2024-04-26T01:09:00.382 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.382 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.382 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.382 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.382 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.341851+0000", 2024-04-26T01:09:00.382 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.382 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.382 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.382 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.382 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.341851+0000", 2024-04-26T01:09:00.383 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.383 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.383 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.383 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.383 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.341852+0000", 2024-04-26T01:09:00.383 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.1299999999999998e-07 2024-04-26T01:09:00.383 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.383 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.383 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.383 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.341874+0000", 2024-04-26T01:09:00.383 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1823000000000001e-05 2024-04-26T01:09:00.383 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.383 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.383 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.383 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.341887+0000", 2024-04-26T01:09:00.384 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3363999999999999e-05 2024-04-26T01:09:00.384 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.384 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.384 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.384 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.341897+0000", 2024-04-26T01:09:00.384 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.6860000000000001e-06 2024-04-26T01:09:00.384 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.384 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.384 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.384 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.348309+0000", 2024-04-26T01:09:00.384 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0064123100000000001 2024-04-26T01:09:00.384 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.384 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.384 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:00.384 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.348455+0000", 2024-04-26T01:09:00.385 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014528100000000001 2024-04-26T01:09:00.385 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.385 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.385 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.385 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.420399+0000", 2024-04-26T01:09:00.385 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.071944662000000006 2024-04-26T01:09:00.385 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.385 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.385 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.385 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.420670+0000", 2024-04-26T01:09:00.385 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00027065900000000003 2024-04-26T01:09:00.385 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.385 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.385 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.386 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:43.424693+0000", 2024-04-26T01:09:00.386 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0040227989999999996 2024-04-26T01:09:00.386 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.386 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.386 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.386 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.386 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.386 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:950 3.cs0 3:3ca9ce7a:::smithi16428812-23:head [copy-from ver 97] snapc 1e=[1d,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-26T01:09:00.386 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:54.058372+0000", 2024-04-26T01:09:00.386 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 6.2340520780000004, 2024-04-26T01:09:00.386 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.028949275, 2024-04-26T01:09:00.386 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.386 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.387 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:00.387 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.387 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.387 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.387 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 950 2024-04-26T01:09:00.387 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.387 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.387 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.387 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.387 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058372+0000", 2024-04-26T01:09:00.387 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.387 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.387 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.387 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.387 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058372+0000", 2024-04-26T01:09:00.388 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.388 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.388 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.388 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.388 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058377+0000", 2024-04-26T01:09:00.388 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.4569999999999998e-06 2024-04-26T01:09:00.388 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.388 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.388 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.388 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058394+0000", 2024-04-26T01:09:00.388 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7249999999999999e-05 2024-04-26T01:09:00.388 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.388 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.388 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.388 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058440+0000", 2024-04-26T01:09:00.389 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.5973999999999997e-05 2024-04-26T01:09:00.389 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.389 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.389 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.389 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058467+0000", 2024-04-26T01:09:00.389 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7025999999999998e-05 2024-04-26T01:09:00.389 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.389 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.389 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.389 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058702+0000", 2024-04-26T01:09:00.389 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000234751 2024-04-26T01:09:00.389 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.389 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.389 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op must wait for map", 2024-04-26T01:09:00.389 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058723+0000", 2024-04-26T01:09:00.390 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1112999999999999e-05 2024-04-26T01:09:00.390 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.390 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.390 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.390 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.062802+0000", 2024-04-26T01:09:00.390 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0040790169999999999 2024-04-26T01:09:00.390 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.390 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.390 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.390 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.062998+0000", 2024-04-26T01:09:00.390 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019606699999999999 2024-04-26T01:09:00.390 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.391 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.391 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:09:00.391 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.078775+0000", 2024-04-26T01:09:00.391 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.015777258999999998 2024-04-26T01:09:00.391 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.391 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.391 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:09:00.391 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.083147+0000", 2024-04-26T01:09:00.391 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0043719880000000003 2024-04-26T01:09:00.391 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.391 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.391 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:09:00.391 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.087082+0000", 2024-04-26T01:09:00.392 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0039349559999999999 2024-04-26T01:09:00.392 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.392 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.392 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.392 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.087321+0000", 2024-04-26T01:09:00.392 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00023941699999999999 2024-04-26T01:09:00.392 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.392 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.392 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.392 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.392 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.392 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:951 3.cs0 3:3ca9ce7a:::smithi16428812-23:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e88)", 2024-04-26T01:09:00.392 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:54.058511+0000", 2024-04-26T01:09:00.392 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 6.2339134740000004, 2024-04-26T01:09:00.392 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.02449397, 2024-04-26T01:09:00.393 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.393 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.393 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:09:00.393 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.393 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.393 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.393 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 951 2024-04-26T01:09:00.393 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.393 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.393 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.393 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.394 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058511+0000", 2024-04-26T01:09:00.394 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.394 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.394 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.394 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.394 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058511+0000", 2024-04-26T01:09:00.394 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.394 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.394 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.394 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.394 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058512+0000", 2024-04-26T01:09:00.394 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.654e-06 2024-04-26T01:09:00.394 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.395 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.395 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.395 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058524+0000", 2024-04-26T01:09:00.395 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.114e-05 2024-04-26T01:09:00.395 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.395 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.395 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.395 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058546+0000", 2024-04-26T01:09:00.395 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2586e-05 2024-04-26T01:09:00.395 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.395 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.395 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.395 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058563+0000", 2024-04-26T01:09:00.395 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7326999999999999e-05 2024-04-26T01:09:00.395 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.396 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.396 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.396 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058898+0000", 2024-04-26T01:09:00.396 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033411399999999999 2024-04-26T01:09:00.396 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.396 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.396 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:09:00.396 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.058918+0000", 2024-04-26T01:09:00.396 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0667000000000001e-05 2024-04-26T01:09:00.396 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.396 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.396 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.396 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.063419+0000", 2024-04-26T01:09:00.396 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.004500418 2024-04-26T01:09:00.396 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.397 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.397 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for blocked object", 2024-04-26T01:09:00.397 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.063572+0000", 2024-04-26T01:09:00.397 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015374400000000001 2024-04-26T01:09:00.397 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.397 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.397 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.397 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.082441+0000", 2024-04-26T01:09:00.397 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.018868598 2024-04-26T01:09:00.397 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.397 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.397 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.397 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.082710+0000", 2024-04-26T01:09:00.397 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00026882099999999998 2024-04-26T01:09:00.398 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.398 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.398 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.398 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:54.083005+0000", 2024-04-26T01:09:00.398 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00029490099999999997 2024-04-26T01:09:00.398 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.398 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.398 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.398 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.398 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.398 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:09:00.400 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:09:00.542 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:09:00.543 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:09:00.543 INFO:teuthology.orchestra.run.smithi012.stdout: "num_ops": 0 2024-04-26T01:09:00.543 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:09:00.558 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:09:00.677 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:09:00.678 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [], 2024-04-26T01:09:00.678 INFO:teuthology.orchestra.run.smithi012.stdout: "complaint_time": 30, 2024-04-26T01:09:00.678 INFO:teuthology.orchestra.run.smithi012.stdout: "num_blocked_ops": 0 2024-04-26T01:09:00.678 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:09:00.694 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:09:00.821 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-26T01:09:00.821 INFO:teuthology.orchestra.run.smithi012.stdout: "size": 20, 2024-04-26T01:09:00.821 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 600, 2024-04-26T01:09:00.821 INFO:teuthology.orchestra.run.smithi012.stdout: "ops": [ 2024-04-26T01:09:00.822 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.822 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-26T01:09:00.822 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:46.050143+0000", 2024-04-26T01:09:00.822 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 74.770805891999998, 2024-04-26T01:09:00.822 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.36180093800000002, 2024-04-26T01:09:00.822 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.822 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.822 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:00.822 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.822 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:09:00.822 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:09:00.822 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 1 2024-04-26T01:09:00.822 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.822 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.823 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.823 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.823 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050143+0000", 2024-04-26T01:09:00.823 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.823 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.823 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.823 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.823 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050143+0000", 2024-04-26T01:09:00.823 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.823 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.823 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.823 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.824 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050146+0000", 2024-04-26T01:09:00.824 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.7829999999999999e-06 2024-04-26T01:09:00.824 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.824 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.824 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.824 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050160+0000", 2024-04-26T01:09:00.824 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4314000000000001e-05 2024-04-26T01:09:00.824 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.824 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.824 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.824 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050182+0000", 2024-04-26T01:09:00.824 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.2407000000000001e-05 2024-04-26T01:09:00.824 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.824 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.825 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.825 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050200+0000", 2024-04-26T01:09:00.825 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7547e-05 2024-04-26T01:09:00.825 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.825 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.825 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.825 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050332+0000", 2024-04-26T01:09:00.825 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013237700000000001 2024-04-26T01:09:00.825 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.825 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.825 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for peered", 2024-04-26T01:09:00.825 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.050361+0000", 2024-04-26T01:09:00.825 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.8704000000000002e-05 2024-04-26T01:09:00.825 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.825 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.826 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.826 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.407786+0000", 2024-04-26T01:09:00.826 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.35742484899999999 2024-04-26T01:09:00.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.826 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.826 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.826 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.408252+0000", 2024-04-26T01:09:00.826 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00046585099999999998 2024-04-26T01:09:00.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.826 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.826 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:09:00.826 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.408586+0000", 2024-04-26T01:09:00.826 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033444200000000001 2024-04-26T01:09:00.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.827 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.827 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:09:00.827 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.409758+0000", 2024-04-26T01:09:00.827 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001171718 2024-04-26T01:09:00.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.827 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.827 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:09:00.827 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.411769+0000", 2024-04-26T01:09:00.827 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0020110520000000002 2024-04-26T01:09:00.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.827 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.827 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:09:00.827 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.411870+0000", 2024-04-26T01:09:00.827 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000101409 2024-04-26T01:09:00.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.828 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.828 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.828 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.411944+0000", 2024-04-26T01:09:00.828 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.3485000000000005e-05 2024-04-26T01:09:00.828 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.828 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.828 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.828 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.828 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:90 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-26T01:09:00.828 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:46.601265+0000", 2024-04-26T01:09:00.828 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 74.219683669999995, 2024-04-26T01:09:00.828 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98632521799999995, 2024-04-26T01:09:00.828 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.829 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.829 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:00.829 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.829 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:09:00.829 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:09:00.829 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 90 2024-04-26T01:09:00.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.829 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.829 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.829 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.829 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601265+0000", 2024-04-26T01:09:00.829 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.829 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.829 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.830 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601265+0000", 2024-04-26T01:09:00.830 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.830 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.830 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.830 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.830 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601267+0000", 2024-04-26T01:09:00.830 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.37e-06 2024-04-26T01:09:00.830 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.830 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.830 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.830 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601418+0000", 2024-04-26T01:09:00.830 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015154400000000001 2024-04-26T01:09:00.830 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.830 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.831 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.831 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601436+0000", 2024-04-26T01:09:00.831 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.7666999999999999e-05 2024-04-26T01:09:00.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.831 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.831 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.831 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.601447+0000", 2024-04-26T01:09:00.831 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1406999999999999e-05 2024-04-26T01:09:00.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.831 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.831 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.831 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.602924+0000", 2024-04-26T01:09:00.831 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0014766899999999999 2024-04-26T01:09:00.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.831 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.832 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.832 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.603071+0000", 2024-04-26T01:09:00.832 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014674600000000001 2024-04-26T01:09:00.832 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.832 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.832 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:09:00.832 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.603406+0000", 2024-04-26T01:09:00.832 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00033568500000000001 2024-04-26T01:09:00.832 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.832 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.832 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:09:00.832 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.604963+0000", 2024-04-26T01:09:00.832 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0015563809999999999 2024-04-26T01:09:00.832 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.833 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.833 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:09:00.833 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587422+0000", 2024-04-26T01:09:00.833 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98245948699999996 2024-04-26T01:09:00.833 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.833 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.833 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:09:00.833 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587520+0000", 2024-04-26T01:09:00.833 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.7650000000000002e-05 2024-04-26T01:09:00.833 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.833 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.833 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.833 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587590+0000", 2024-04-26T01:09:00.833 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.0591000000000002e-05 2024-04-26T01:09:00.833 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.834 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.834 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.834 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.834 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.834 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_repop_reply(client.4188.0:90 1.0 e24/23 ondisk, result = 0)", 2024-04-26T01:09:00.834 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:46.605376+0000", 2024-04-26T01:09:00.834 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 74.215573269999993, 2024-04-26T01:09:00.834 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98225920899999997, 2024-04-26T01:09:00.834 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.834 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.834 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:09:00.834 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.834 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.834 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.835 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605376+0000", 2024-04-26T01:09:00.835 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.835 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.835 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.835 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.835 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605376+0000", 2024-04-26T01:09:00.835 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.835 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.835 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.835 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.835 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605377+0000", 2024-04-26T01:09:00.835 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.407e-06 2024-04-26T01:09:00.835 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.835 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.835 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.836 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:46.605384+0000", 2024-04-26T01:09:00.836 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.5299999999999999e-06 2024-04-26T01:09:00.836 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.836 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.836 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.836 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587134+0000", 2024-04-26T01:09:00.836 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.98174979699999998 2024-04-26T01:09:00.836 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.836 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.836 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.836 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587156+0000", 2024-04-26T01:09:00.836 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.1704000000000001e-05 2024-04-26T01:09:00.836 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.836 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.837 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.837 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587391+0000", 2024-04-26T01:09:00.837 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000235052 2024-04-26T01:09:00.837 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.837 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.837 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.837 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587412+0000", 2024-04-26T01:09:00.837 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0630999999999999e-05 2024-04-26T01:09:00.837 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.837 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.837 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.837 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.587635+0000", 2024-04-26T01:09:00.837 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000223088 2024-04-26T01:09:00.837 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.837 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.838 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.838 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.838 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.838 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4188.0:145 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-26T01:09:00.838 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:47.981007+0000", 2024-04-26T01:09:00.838 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 72.839941408000001, 2024-04-26T01:09:00.838 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.057997620999999999, 2024-04-26T01:09:00.838 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.838 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.838 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:00.838 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.838 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4188", 2024-04-26T01:09:00.838 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/2483988485", 2024-04-26T01:09:00.838 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 145 2024-04-26T01:09:00.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.839 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.839 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.839 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.839 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981007+0000", 2024-04-26T01:09:00.839 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.839 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.839 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.839 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981007+0000", 2024-04-26T01:09:00.839 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.839 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.839 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.839 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981009+0000", 2024-04-26T01:09:00.840 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.204e-06 2024-04-26T01:09:00.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.840 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.840 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.840 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981146+0000", 2024-04-26T01:09:00.840 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00013786000000000001 2024-04-26T01:09:00.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.840 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.840 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.840 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981165+0000", 2024-04-26T01:09:00.840 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8542999999999998e-05 2024-04-26T01:09:00.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.840 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.840 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.840 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.981177+0000", 2024-04-26T01:09:00.841 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1787000000000001e-05 2024-04-26T01:09:00.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.841 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.841 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.841 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.982190+0000", 2024-04-26T01:09:00.841 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0010130180000000001 2024-04-26T01:09:00.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.841 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.841 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.841 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.982236+0000", 2024-04-26T01:09:00.841 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.6445e-05 2024-04-26T01:09:00.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.841 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.841 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for subops from 2", 2024-04-26T01:09:00.842 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.982311+0000", 2024-04-26T01:09:00.842 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.4657e-05 2024-04-26T01:09:00.842 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.842 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.842 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "op_commit", 2024-04-26T01:09:00.842 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:47.983568+0000", 2024-04-26T01:09:00.842 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001257172 2024-04-26T01:09:00.842 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.842 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.842 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:09:00.842 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.038854+0000", 2024-04-26T01:09:00.842 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.055285918000000003 2024-04-26T01:09:00.842 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.842 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.843 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:09:00.843 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.038941+0000", 2024-04-26T01:09:00.843 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.6971999999999993e-05 2024-04-26T01:09:00.843 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.843 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.843 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.843 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:48.039005+0000", 2024-04-26T01:09:00.843 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.4045000000000001e-05 2024-04-26T01:09:00.843 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.843 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.843 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.843 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.843 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.843 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:33 3.2s0 3:410d186f:::smithi16428812-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:09:00.844 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.328026+0000", 2024-04-26T01:09:00.844 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 61.492923060999999, 2024-04-26T01:09:00.844 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.065568274999999995, 2024-04-26T01:09:00.844 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.844 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.844 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:09:00.844 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.844 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.844 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.844 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 33 2024-04-26T01:09:00.844 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.844 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.844 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.844 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.844 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328026+0000", 2024-04-26T01:09:00.845 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.845 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.845 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.845 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.845 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328026+0000", 2024-04-26T01:09:00.845 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.845 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.845 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.845 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.845 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328028+0000", 2024-04-26T01:09:00.845 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.357e-06 2024-04-26T01:09:00.845 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.845 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.845 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.845 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328040+0000", 2024-04-26T01:09:00.846 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2257e-05 2024-04-26T01:09:00.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.846 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.846 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.846 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328089+0000", 2024-04-26T01:09:00.846 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.8164000000000002e-05 2024-04-26T01:09:00.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.846 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.846 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.846 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328112+0000", 2024-04-26T01:09:00.846 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3444000000000001e-05 2024-04-26T01:09:00.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.846 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.846 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.847 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328852+0000", 2024-04-26T01:09:00.847 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00074019899999999998 2024-04-26T01:09:00.847 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.847 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.847 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:00.847 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.328981+0000", 2024-04-26T01:09:00.847 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012923500000000001 2024-04-26T01:09:00.847 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.847 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.847 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.847 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.339980+0000", 2024-04-26T01:09:00.847 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.010998232 2024-04-26T01:09:00.847 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.847 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.848 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.848 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.340144+0000", 2024-04-26T01:09:00.848 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016476699999999999 2024-04-26T01:09:00.848 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.848 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.848 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.848 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.393594+0000", 2024-04-26T01:09:00.848 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.053449620000000003 2024-04-26T01:09:00.848 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.848 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.848 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.848 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.848 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.848 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 33/30 ECSubWriteReply(tid=162, last_complete=33'4, committed=1, applied=1))", 2024-04-26T01:09:00.848 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.341290+0000", 2024-04-26T01:09:00.849 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 61.479659292000001, 2024-04-26T01:09:00.849 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.050669963999999998, 2024-04-26T01:09:00.849 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.849 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.849 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:09:00.849 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.849 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.849 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.849 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.341290+0000", 2024-04-26T01:09:00.849 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.849 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.849 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.849 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.849 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.341290+0000", 2024-04-26T01:09:00.849 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.850 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.850 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.850 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.850 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.341291+0000", 2024-04-26T01:09:00.850 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.237e-06 2024-04-26T01:09:00.850 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.850 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.850 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.850 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.341301+0000", 2024-04-26T01:09:00.850 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0266e-05 2024-04-26T01:09:00.850 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.850 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.850 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.850 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.388586+0000", 2024-04-26T01:09:00.851 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.047284633 2024-04-26T01:09:00.851 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.851 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.851 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.851 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.388599+0000", 2024-04-26T01:09:00.851 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3335e-05 2024-04-26T01:09:00.851 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.851 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.851 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.851 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.391887+0000", 2024-04-26T01:09:00.851 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0032881400000000002 2024-04-26T01:09:00.851 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.851 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.851 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.852 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.391959+0000", 2024-04-26T01:09:00.852 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.2353000000000003e-05 2024-04-26T01:09:00.852 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.852 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.852 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.852 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.852 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.852 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:81 3.2s0 3:4b558697:::smithi16428812-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:00.852 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.385353+0000", 2024-04-26T01:09:00.852 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 61.435595773000003, 2024-04-26T01:09:00.852 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.12263547800000001, 2024-04-26T01:09:00.852 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.852 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.852 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:09:00.852 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.853 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.853 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.853 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 81 2024-04-26T01:09:00.853 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.853 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.853 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.853 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.853 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385353+0000", 2024-04-26T01:09:00.853 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.853 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.853 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.853 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.853 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385353+0000", 2024-04-26T01:09:00.853 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.854 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.854 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.854 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.854 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385354+0000", 2024-04-26T01:09:00.854 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.6300000000000004e-07 2024-04-26T01:09:00.854 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.854 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.854 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.854 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385357+0000", 2024-04-26T01:09:00.854 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.4989999999999999e-06 2024-04-26T01:09:00.854 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.854 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.854 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.855 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385381+0000", 2024-04-26T01:09:00.855 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3643999999999999e-05 2024-04-26T01:09:00.855 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.855 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.855 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.855 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.385390+0000", 2024-04-26T01:09:00.855 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.9090000000000006e-06 2024-04-26T01:09:00.855 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.855 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.855 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.855 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.392442+0000", 2024-04-26T01:09:00.855 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.007052455 2024-04-26T01:09:00.855 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.855 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.855 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:00.856 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.392642+0000", 2024-04-26T01:09:00.856 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000199662 2024-04-26T01:09:00.856 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.856 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.856 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.856 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.436978+0000", 2024-04-26T01:09:00.856 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.044336042999999999 2024-04-26T01:09:00.856 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.856 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.856 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.856 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.437074+0000", 2024-04-26T01:09:00.856 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5700999999999997e-05 2024-04-26T01:09:00.856 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.856 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.857 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.857 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.507989+0000", 2024-04-26T01:09:00.857 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.070914701999999996 2024-04-26T01:09:00.857 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.857 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.857 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.857 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.857 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.857 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpReadReply(3.2s0 33/30 ECSubReadReply(tid=169, attrs_read=0))", 2024-04-26T01:09:00.857 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.443824+0000", 2024-04-26T01:09:00.857 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 61.377124713000001, 2024-04-26T01:09:00.857 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.064178733000000002, 2024-04-26T01:09:00.857 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.857 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.857 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:09:00.858 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.858 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.858 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.858 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.443824+0000", 2024-04-26T01:09:00.858 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.858 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.858 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.858 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.858 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.443824+0000", 2024-04-26T01:09:00.858 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.858 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.858 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.858 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.858 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.443825+0000", 2024-04-26T01:09:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.266e-06 2024-04-26T01:09:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.443845+0000", 2024-04-26T01:09:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.9236e-05 2024-04-26T01:09:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.505240+0000", 2024-04-26T01:09:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.061395523 2024-04-26T01:09:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.505256+0000", 2024-04-26T01:09:00.860 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6101999999999999e-05 2024-04-26T01:09:00.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.860 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.860 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.860 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.507809+0000", 2024-04-26T01:09:00.860 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0025529820000000001 2024-04-26T01:09:00.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.860 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.860 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.860 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508003+0000", 2024-04-26T01:09:00.860 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00019362399999999999 2024-04-26T01:09:00.860 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.860 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.860 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.861 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.861 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:166 3.2s0 3:410d186f:::smithi16428812-11:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:00.861 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.504084+0000", 2024-04-26T01:09:00.861 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 61.316864608000003, 2024-04-26T01:09:00.861 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.062718093000000003, 2024-04-26T01:09:00.861 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.861 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.861 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:00.861 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.861 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.861 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.861 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 166 2024-04-26T01:09:00.861 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.861 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.861 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.862 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.862 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504084+0000", 2024-04-26T01:09:00.862 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.862 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.862 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.862 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.862 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504084+0000", 2024-04-26T01:09:00.862 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.862 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.862 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.862 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.862 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504086+0000", 2024-04-26T01:09:00.862 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8819999999999999e-06 2024-04-26T01:09:00.862 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.863 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.863 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.863 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504689+0000", 2024-04-26T01:09:00.863 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000602634 2024-04-26T01:09:00.863 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.863 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.863 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.863 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504716+0000", 2024-04-26T01:09:00.863 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.6995000000000002e-05 2024-04-26T01:09:00.863 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.863 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.863 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.863 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504732+0000", 2024-04-26T01:09:00.863 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.5896000000000002e-05 2024-04-26T01:09:00.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.864 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.864 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.864 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504855+0000", 2024-04-26T01:09:00.864 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012368000000000001 2024-04-26T01:09:00.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.864 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.864 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.864 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.505004+0000", 2024-04-26T01:09:00.864 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000148515 2024-04-26T01:09:00.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.864 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.864 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:09:00.864 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.506133+0000", 2024-04-26T01:09:00.865 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001129625 2024-04-26T01:09:00.865 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.865 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.865 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:09:00.865 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508806+0000", 2024-04-26T01:09:00.865 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0026721760000000001 2024-04-26T01:09:00.865 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.865 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.865 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:09:00.865 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566633+0000", 2024-04-26T01:09:00.865 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.057827207999999998 2024-04-26T01:09:00.865 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.865 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.865 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.865 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566802+0000", 2024-04-26T01:09:00.866 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00016948199999999999 2024-04-26T01:09:00.866 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.866 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.866 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.866 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.866 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.866 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:167 3.2s0 3:410d186f:::smithi16428812-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:00.866 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.504769+0000", 2024-04-26T01:09:00.866 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 61.316180164999999, 2024-04-26T01:09:00.866 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.18953119900000001, 2024-04-26T01:09:00.866 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.866 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.866 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:00.866 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.867 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.867 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.867 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 167 2024-04-26T01:09:00.867 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.867 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.867 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.867 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.867 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504769+0000", 2024-04-26T01:09:00.867 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.867 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.867 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.867 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.867 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504769+0000", 2024-04-26T01:09:00.867 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.867 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504770+0000", 2024-04-26T01:09:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.356e-06 2024-04-26T01:09:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504777+0000", 2024-04-26T01:09:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.8140000000000003e-06 2024-04-26T01:09:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504791+0000", 2024-04-26T01:09:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.4267000000000001e-05 2024-04-26T01:09:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.869 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.869 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.869 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504807+0000", 2024-04-26T01:09:00.869 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6030000000000001e-05 2024-04-26T01:09:00.869 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.869 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.869 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.869 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.506957+0000", 2024-04-26T01:09:00.869 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0021496549999999999 2024-04-26T01:09:00.869 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.869 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.869 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.869 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.507039+0000", 2024-04-26T01:09:00.869 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.1959000000000006e-05 2024-04-26T01:09:00.869 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.870 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.870 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:09:00.870 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.507462+0000", 2024-04-26T01:09:00.870 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00042285800000000002 2024-04-26T01:09:00.870 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.870 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.870 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:09:00.870 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.511679+0000", 2024-04-26T01:09:00.870 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0042174209999999998 2024-04-26T01:09:00.870 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.870 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.870 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:09:00.870 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694149+0000", 2024-04-26T01:09:00.870 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.18246974999999999 2024-04-26T01:09:00.871 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.871 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.871 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.871 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694300+0000", 2024-04-26T01:09:00.871 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000151089 2024-04-26T01:09:00.871 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.871 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.871 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.871 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.871 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.871 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:168 3.2s0 3:410d186f:::smithi16428812-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:00.871 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.504832+0000", 2024-04-26T01:09:00.871 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 61.316116714000003, 2024-04-26T01:09:00.871 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.19389620699999999, 2024-04-26T01:09:00.871 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.872 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.872 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:09:00.872 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.872 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.872 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.872 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 168 2024-04-26T01:09:00.872 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.872 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.872 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.872 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.872 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504832+0000", 2024-04-26T01:09:00.872 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.872 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.872 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.873 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.873 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504832+0000", 2024-04-26T01:09:00.873 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.873 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.873 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.873 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.873 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504833+0000", 2024-04-26T01:09:00.873 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.5199999999999995e-07 2024-04-26T01:09:00.873 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.873 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.873 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.873 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504838+0000", 2024-04-26T01:09:00.873 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.1560000000000003e-06 2024-04-26T01:09:00.873 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.873 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.874 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.874 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504853+0000", 2024-04-26T01:09:00.874 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.469e-05 2024-04-26T01:09:00.874 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.874 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.874 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.874 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.504865+0000", 2024-04-26T01:09:00.874 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.1707e-05 2024-04-26T01:09:00.874 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.874 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.874 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.874 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508279+0000", 2024-04-26T01:09:00.874 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0034144650000000002 2024-04-26T01:09:00.874 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.875 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.875 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:00.875 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508350+0000", 2024-04-26T01:09:00.875 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1166000000000003e-05 2024-04-26T01:09:00.875 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.875 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.875 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.875 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694397+0000", 2024-04-26T01:09:00.875 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.18604716299999999 2024-04-26T01:09:00.875 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.875 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.875 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.875 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694544+0000", 2024-04-26T01:09:00.875 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014672200000000001 2024-04-26T01:09:00.876 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.876 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.876 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.876 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.698728+0000", 2024-04-26T01:09:00.876 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0041841860000000003 2024-04-26T01:09:00.876 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.876 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.876 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.876 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.876 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.876 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 33/30 ECSubWriteReply(tid=173, last_complete=33'8, committed=1, applied=1))", 2024-04-26T01:09:00.876 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.508639+0000", 2024-04-26T01:09:00.876 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 61.312309835000001, 2024-04-26T01:09:00.876 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.058189232, 2024-04-26T01:09:00.877 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.877 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.877 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:09:00.877 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.877 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.877 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.877 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508639+0000", 2024-04-26T01:09:00.877 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.877 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.877 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.877 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.877 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508639+0000", 2024-04-26T01:09:00.877 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.877 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.878 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.878 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.878 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508640+0000", 2024-04-26T01:09:00.878 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.9999999999999997e-07 2024-04-26T01:09:00.878 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.878 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.878 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.878 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.508646+0000", 2024-04-26T01:09:00.878 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.0580000000000003e-06 2024-04-26T01:09:00.878 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.878 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.878 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.878 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566182+0000", 2024-04-26T01:09:00.878 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.057536449000000003 2024-04-26T01:09:00.879 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.879 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.879 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.879 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566201+0000", 2024-04-26T01:09:00.879 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8904000000000001e-05 2024-04-26T01:09:00.879 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.879 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.879 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.879 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566315+0000", 2024-04-26T01:09:00.879 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00011435800000000001 2024-04-26T01:09:00.879 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.879 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.879 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.879 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566828+0000", 2024-04-26T01:09:00.880 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00051276299999999996 2024-04-26T01:09:00.880 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.880 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.880 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.880 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.880 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.880 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:191 3.2s0 3:4b558697:::smithi16428812-27:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:00.880 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.563295+0000", 2024-04-26T01:09:00.880 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 61.257653900000001, 2024-04-26T01:09:00.880 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.13397727300000001, 2024-04-26T01:09:00.880 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.880 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.880 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:00.880 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.880 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.881 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.881 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 191 2024-04-26T01:09:00.881 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.881 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.881 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.881 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.881 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563295+0000", 2024-04-26T01:09:00.881 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.881 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.881 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.881 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.881 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563295+0000", 2024-04-26T01:09:00.881 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.881 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.882 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.882 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.882 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563297+0000", 2024-04-26T01:09:00.882 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4140000000000001e-06 2024-04-26T01:09:00.882 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.882 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.882 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.882 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563318+0000", 2024-04-26T01:09:00.882 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.0211e-05 2024-04-26T01:09:00.882 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.882 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.882 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.882 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563342+0000", 2024-04-26T01:09:00.882 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.4266000000000001e-05 2024-04-26T01:09:00.883 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.883 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.883 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.883 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563353+0000", 2024-04-26T01:09:00.883 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0886e-05 2024-04-26T01:09:00.883 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.883 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.883 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.883 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563479+0000", 2024-04-26T01:09:00.883 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000126651 2024-04-26T01:09:00.883 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.883 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.883 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.883 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.563655+0000", 2024-04-26T01:09:00.883 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000175713 2024-04-26T01:09:00.884 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.884 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.884 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:09:00.884 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.564447+0000", 2024-04-26T01:09:00.884 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00079165500000000005 2024-04-26T01:09:00.884 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.884 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.884 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:09:00.884 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.565861+0000", 2024-04-26T01:09:00.884 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001414595 2024-04-26T01:09:00.884 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.884 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.884 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:09:00.884 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.697131+0000", 2024-04-26T01:09:00.885 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.13126945700000001 2024-04-26T01:09:00.885 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.885 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.885 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.885 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.697272+0000", 2024-04-26T01:09:00.885 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00014142500000000001 2024-04-26T01:09:00.885 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.885 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.885 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.885 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.885 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.885 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 33/30 ECSubWriteReply(tid=174, last_complete=33'9, committed=1, applied=1))", 2024-04-26T01:09:00.885 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:07:59.566047+0000", 2024-04-26T01:09:00.885 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 61.254901809000003, 2024-04-26T01:09:00.885 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.128277063, 2024-04-26T01:09:00.886 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.886 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.886 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:09:00.886 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.886 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.886 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.886 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566047+0000", 2024-04-26T01:09:00.886 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.886 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.886 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.886 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.886 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566047+0000", 2024-04-26T01:09:00.886 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.886 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.887 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.887 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.887 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566048+0000", 2024-04-26T01:09:00.887 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.316e-06 2024-04-26T01:09:00.887 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.887 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.887 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.887 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.566138+0000", 2024-04-26T01:09:00.887 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.9281000000000001e-05 2024-04-26T01:09:00.887 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.887 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.887 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.887 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.693854+0000", 2024-04-26T01:09:00.887 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.12771605 2024-04-26T01:09:00.887 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.888 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.888 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.888 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.693868+0000", 2024-04-26T01:09:00.888 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3910000000000001e-05 2024-04-26T01:09:00.888 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.888 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.888 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.888 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694017+0000", 2024-04-26T01:09:00.888 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000149865 2024-04-26T01:09:00.888 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.888 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.888 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.888 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:07:59.694324+0000", 2024-04-26T01:09:00.888 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00030664100000000002 2024-04-26T01:09:00.888 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.889 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.889 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.889 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.889 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.889 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 38/30 ECSubWriteReply(tid=190, last_complete=38'7, committed=1, applied=1))", 2024-04-26T01:09:00.889 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:04.497964+0000", 2024-04-26T01:09:00.889 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 56.322984489, 2024-04-26T01:09:00.889 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.81206265399999999, 2024-04-26T01:09:00.889 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.889 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.889 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:09:00.889 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.889 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.889 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.890 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.497964+0000", 2024-04-26T01:09:00.890 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.890 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.890 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.890 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.890 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.497964+0000", 2024-04-26T01:09:00.890 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.890 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.890 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.890 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.890 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.497966+0000", 2024-04-26T01:09:00.890 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.274e-06 2024-04-26T01:09:00.890 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.890 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.890 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.891 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:04.497979+0000", 2024-04-26T01:09:00.891 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3127e-05 2024-04-26T01:09:00.891 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.891 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.891 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.891 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.309733+0000", 2024-04-26T01:09:00.891 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.81175416700000003 2024-04-26T01:09:00.891 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.891 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.891 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.891 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.309746+0000", 2024-04-26T01:09:00.891 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.3523000000000001e-05 2024-04-26T01:09:00.891 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.891 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.891 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.892 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.309899+0000", 2024-04-26T01:09:00.892 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00015288199999999999 2024-04-26T01:09:00.892 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.892 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.892 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.892 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:05.310027+0000", 2024-04-26T01:09:00.892 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012768100000000001 2024-04-26T01:09:00.892 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.892 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.892 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.892 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.892 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.892 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "MOSDECSubOpWrite(3.fs3 50/44 ECSubWrite(tid=225, reqid=client.4521.0:494, at_version=50'41, trim_to=0'0, roll_forward_to=50'38))", 2024-04-26T01:09:00.892 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:16.091583+0000", 2024-04-26T01:09:00.893 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 44.729365864000002, 2024-04-26T01:09:00.893 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.14404377500000001, 2024-04-26T01:09:00.893 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.893 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.893 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "reached pg", 2024-04-26T01:09:00.893 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.893 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.893 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.893 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.091583+0000", 2024-04-26T01:09:00.893 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.893 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.893 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.893 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.893 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.091583+0000", 2024-04-26T01:09:00.893 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.894 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.894 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.894 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.894 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.091585+0000", 2024-04-26T01:09:00.894 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.075e-06 2024-04-26T01:09:00.894 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.894 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.894 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.894 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.091712+0000", 2024-04-26T01:09:00.894 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012731099999999999 2024-04-26T01:09:00.894 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.894 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.894 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.894 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.233513+0000", 2024-04-26T01:09:00.895 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.14180088399999999 2024-04-26T01:09:00.895 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.895 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.895 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.895 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.233530+0000", 2024-04-26T01:09:00.895 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6390000000000001e-05 2024-04-26T01:09:00.895 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.895 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.895 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.895 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.233655+0000", 2024-04-26T01:09:00.895 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000125341 2024-04-26T01:09:00.895 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.895 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.895 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:09:00.895 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.233671+0000", 2024-04-26T01:09:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.6515999999999999e-05 2024-04-26T01:09:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:09:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.235542+0000", 2024-04-26T01:09:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.001870125 2024-04-26T01:09:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:16.235627+0000", 2024-04-26T01:09:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.5133e-05 2024-04-26T01:09:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.897 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.897 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:741 3.2s0 3:49548768:::smithi16428812-5:head [setxattr _header (56) in=7b] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:09:00.897 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:37.149865+0000", 2024-04-26T01:09:00.897 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 23.671084123, 2024-04-26T01:09:00.897 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.165434726, 2024-04-26T01:09:00.897 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.897 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.897 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:00.897 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.897 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.897 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.897 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 741 2024-04-26T01:09:00.897 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.897 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.897 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.898 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.898 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.149865+0000", 2024-04-26T01:09:00.898 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.898 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.898 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.898 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.898 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.149865+0000", 2024-04-26T01:09:00.898 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.898 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.898 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.898 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.898 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.149866+0000", 2024-04-26T01:09:00.898 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.0299999999999997e-07 2024-04-26T01:09:00.898 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.898 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.899 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.899 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.149871+0000", 2024-04-26T01:09:00.899 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.7799999999999997e-06 2024-04-26T01:09:00.899 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.899 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.899 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.899 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.149881+0000", 2024-04-26T01:09:00.899 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.0051000000000001e-05 2024-04-26T01:09:00.899 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.899 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.899 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.899 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.149891+0000", 2024-04-26T01:09:00.899 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 9.8060000000000006e-06 2024-04-26T01:09:00.899 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.899 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.900 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.900 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.151680+0000", 2024-04-26T01:09:00.900 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0017883280000000001 2024-04-26T01:09:00.900 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.900 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.900 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.900 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.151791+0000", 2024-04-26T01:09:00.900 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000111167 2024-04-26T01:09:00.900 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.900 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.900 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_started", 2024-04-26T01:09:00.900 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.152338+0000", 2024-04-26T01:09:00.900 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00054768000000000004 2024-04-26T01:09:00.900 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.900 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.901 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "sub_op_committed", 2024-04-26T01:09:00.901 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.154661+0000", 2024-04-26T01:09:00.901 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0023228950000000002 2024-04-26T01:09:00.901 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.901 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.901 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "commit_sent", 2024-04-26T01:09:00.901 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.315211+0000", 2024-04-26T01:09:00.901 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.160549948 2024-04-26T01:09:00.901 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.901 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.901 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.901 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.315299+0000", 2024-04-26T01:09:00.901 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 8.8168e-05 2024-04-26T01:09:00.901 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.901 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.902 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.902 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.902 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.902 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:742 3.2s0 3:49548768:::smithi16428812-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:09:00.902 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:37.149907+0000", 2024-04-26T01:09:00.902 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 23.671042286999999, 2024-04-26T01:09:00.902 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.55288251099999997, 2024-04-26T01:09:00.902 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.902 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.902 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:09:00.902 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.902 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.902 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.902 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 742 2024-04-26T01:09:00.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.903 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.903 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.903 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.903 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.149907+0000", 2024-04-26T01:09:00.903 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.903 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.903 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.903 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.149907+0000", 2024-04-26T01:09:00.903 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.903 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.903 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.903 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.149907+0000", 2024-04-26T01:09:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8000000000000005e-07 2024-04-26T01:09:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.149911+0000", 2024-04-26T01:09:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 3.4139999999999998e-06 2024-04-26T01:09:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.149923+0000", 2024-04-26T01:09:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.2533e-05 2024-04-26T01:09:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.149931+0000", 2024-04-26T01:09:00.905 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.9629999999999998e-06 2024-04-26T01:09:00.905 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.905 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.905 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.905 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.153185+0000", 2024-04-26T01:09:00.905 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0032533990000000001 2024-04-26T01:09:00.905 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.905 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.905 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:00.905 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.153300+0000", 2024-04-26T01:09:00.905 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000115812 2024-04-26T01:09:00.905 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.905 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.905 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.315802+0000", 2024-04-26T01:09:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.16250130500000001 2024-04-26T01:09:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.315975+0000", 2024-04-26T01:09:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00017350800000000001 2024-04-26T01:09:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.702789+0000", 2024-04-26T01:09:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.38681379700000001 2024-04-26T01:09:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.907 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.907 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(client.4521.0:757 3.2s0 3:4b558697:::smithi16428812-27:head [read 0~311296,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:09:00.907 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:37.204431+0000", 2024-04-26T01:09:00.907 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 23.616518172999999, 2024-04-26T01:09:00.907 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.10993360000000001, 2024-04-26T01:09:00.907 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.907 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.907 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:09:00.907 INFO:teuthology.orchestra.run.smithi012.stdout: "client_info": { 2024-04-26T01:09:00.907 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4521", 2024-04-26T01:09:00.907 INFO:teuthology.orchestra.run.smithi012.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:00.907 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 757 2024-04-26T01:09:00.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.907 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.908 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.908 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.908 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.204431+0000", 2024-04-26T01:09:00.908 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.908 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.908 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.908 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.908 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.204431+0000", 2024-04-26T01:09:00.908 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.908 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.908 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.908 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.908 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.204431+0000", 2024-04-26T01:09:00.908 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.8800000000000002e-07 2024-04-26T01:09:00.908 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.909 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.909 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.909 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.204437+0000", 2024-04-26T01:09:00.909 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 5.2599999999999996e-06 2024-04-26T01:09:00.909 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.909 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.909 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.909 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.204447+0000", 2024-04-26T01:09:00.909 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.078e-05 2024-04-26T01:09:00.909 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.909 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.909 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.909 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.204455+0000", 2024-04-26T01:09:00.909 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.1430000000000001e-06 2024-04-26T01:09:00.909 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.910 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.910 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.910 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.206233+0000", 2024-04-26T01:09:00.910 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.0017788179999999999 2024-04-26T01:09:00.910 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.910 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.910 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.910 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.206356+0000", 2024-04-26T01:09:00.910 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.00012260099999999999 2024-04-26T01:09:00.910 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.910 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.910 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.910 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:37.314364+0000", 2024-04-26T01:09:00.910 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.10800820999999999 2024-04-26T01:09:00.911 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.911 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.911 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.911 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.911 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.911 INFO:teuthology.orchestra.run.smithi012.stdout: "description": "osd_op(osd.2.21:4 3.2s0 3:49548768:::smithi16428812-5:head [assert-version v32,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:09:00.911 INFO:teuthology.orchestra.run.smithi012.stdout: "initiated_at": "2024-04-26T01:08:47.491740+0000", 2024-04-26T01:09:00.911 INFO:teuthology.orchestra.run.smithi012.stdout: "age": 13.329208919999999, 2024-04-26T01:09:00.911 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.85647151899999996, 2024-04-26T01:09:00.911 INFO:teuthology.orchestra.run.smithi012.stdout: "continuous": false, 2024-04-26T01:09:00.911 INFO:teuthology.orchestra.run.smithi012.stdout: "type_data": { 2024-04-26T01:09:00.911 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "started", 2024-04-26T01:09:00.911 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-26T01:09:00.911 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.911 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated", 2024-04-26T01:09:00.912 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.491740+0000", 2024-04-26T01:09:00.912 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.912 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.912 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.912 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read", 2024-04-26T01:09:00.912 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.491740+0000", 2024-04-26T01:09:00.912 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0 2024-04-26T01:09:00.912 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.912 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.912 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled", 2024-04-26T01:09:00.912 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.491745+0000", 2024-04-26T01:09:00.912 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 4.6360000000000002e-06 2024-04-26T01:09:00.912 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.912 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.912 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read", 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.491763+0000", 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 1.8467000000000001e-05 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched", 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.491835+0000", 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 7.2358000000000001e-05 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "queued_for_pg", 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.491858+0000", 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 2.3003e-05 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.913 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.492019+0000", 2024-04-26T01:09:00.914 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000160524 2024-04-26T01:09:00.914 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.914 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.914 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "waiting for peered", 2024-04-26T01:09:00.914 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:47.492025+0000", 2024-04-26T01:09:00.914 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 6.0739999999999998e-06 2024-04-26T01:09:00.914 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.914 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.914 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "reached_pg", 2024-04-26T01:09:00.914 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:48.333852+0000", 2024-04-26T01:09:00.914 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.84182718400000001 2024-04-26T01:09:00.914 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.914 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.914 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "started", 2024-04-26T01:09:00.915 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:48.334550+0000", 2024-04-26T01:09:00.915 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.000698224 2024-04-26T01:09:00.915 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-26T01:09:00.915 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-26T01:09:00.915 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "done", 2024-04-26T01:09:00.915 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-26T01:08:48.348211+0000", 2024-04-26T01:09:00.915 INFO:teuthology.orchestra.run.smithi012.stdout: "duration": 0.013661049 2024-04-26T01:09:00.915 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.915 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.915 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.915 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-26T01:09:00.915 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-26T01:09:00.915 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-26T01:09:00.919 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:09:01.051 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:09:01.051 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:09:01.052 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:09:01.052 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:09:01.065 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:09:01.181 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:09:01.182 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:09:01.182 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:09:01.182 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:09:01.182 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:09:01.196 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:09:01.322 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:09:01.322 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:09:01.322 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:09:01.322 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:09:01.322 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.322 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_repop(client.4188.0:145 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'103, mlcod=25'98)", 2024-04-26T01:09:01.322 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:47.983085+0000", 2024-04-26T01:09:01.322 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 73.337184902000004, 2024-04-26T01:09:01.322 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.055794923000000003, 2024-04-26T01:09:01.322 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.323 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.323 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:01.323 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.323 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.323 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.323 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983085+0000", 2024-04-26T01:09:01.323 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.323 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.323 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.323 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983085+0000", 2024-04-26T01:09:01.323 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.323 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.324 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.324 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983086+0000", 2024-04-26T01:09:01.324 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9699999999999994e-07 2024-04-26T01:09:01.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.327 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.327 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.327 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:47.983194+0000", 2024-04-26T01:09:01.328 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107569 2024-04-26T01:09:01.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.328 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.328 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.328 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037255+0000", 2024-04-26T01:09:01.328 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.054061302999999998 2024-04-26T01:09:01.329 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.329 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.329 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.329 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037268+0000", 2024-04-26T01:09:01.329 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3477e-05 2024-04-26T01:09:01.330 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.330 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.330 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.330 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037378+0000", 2024-04-26T01:09:01.330 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000110089 2024-04-26T01:09:01.330 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.331 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.331 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:01.331 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.037414+0000", 2024-04-26T01:09:01.331 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5123000000000002e-05 2024-04-26T01:09:01.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.331 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.332 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:09:01.332 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.038790+0000", 2024-04-26T01:09:01.332 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0013768820000000001 2024-04-26T01:09:01.332 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.332 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.332 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.333 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:48.038880+0000", 2024-04-26T01:09:01.333 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9482999999999997e-05 2024-04-26T01:09:01.333 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.333 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.333 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.334 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.334 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpRead(3.2s1 33/30 ECSubRead(tid=163, to_read={3:410d186f:::smithi16428812-11:head=0,4096,0}, subchunks={3:410d186f:::smithi16428812-11:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:09:01.334 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.340790+0000", 2024-04-26T01:09:01.334 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 61.979479904999998, 2024-04-26T01:09:01.334 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.047962901000000002, 2024-04-26T01:09:01.334 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.335 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.335 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.335 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.335 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.335 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.335 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340790+0000", 2024-04-26T01:09:01.335 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.335 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.336 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.336 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.336 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340790+0000", 2024-04-26T01:09:01.336 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.336 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.336 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.336 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340791+0000", 2024-04-26T01:09:01.336 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.3300000000000001e-07 2024-04-26T01:09:01.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.336 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.336 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.336 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.340801+0000", 2024-04-26T01:09:01.337 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0308e-05 2024-04-26T01:09:01.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.337 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.337 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388316+0000", 2024-04-26T01:09:01.337 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.047514702999999998 2024-04-26T01:09:01.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.337 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.338 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388324+0000", 2024-04-26T01:09:01.338 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.3529999999999995e-06 2024-04-26T01:09:01.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.338 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.338 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.338 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388439+0000", 2024-04-26T01:09:01.338 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000114657 2024-04-26T01:09:01.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.338 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.338 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.339 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.388753+0000", 2024-04-26T01:09:01.339 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00031404700000000002 2024-04-26T01:09:01.339 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.339 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.339 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.339 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.339 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:62 3.es0 3:7eef64f7:::smithi16428812-21:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:01.339 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.361875+0000", 2024-04-26T01:09:01.339 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 61.958394831, 2024-04-26T01:09:01.339 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.111184554, 2024-04-26T01:09:01.339 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.340 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.340 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:01.340 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:01.340 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:01.340 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:01.340 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 62 2024-04-26T01:09:01.340 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.340 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.340 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.340 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.340 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361875+0000", 2024-04-26T01:09:01.341 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.341 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.341 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.341 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361875+0000", 2024-04-26T01:09:01.341 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.341 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.341 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.341 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361876+0000", 2024-04-26T01:09:01.342 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5899999999999995e-07 2024-04-26T01:09:01.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.342 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.342 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.342 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361880+0000", 2024-04-26T01:09:01.342 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8739999999999997e-06 2024-04-26T01:09:01.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.342 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.342 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.342 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361887+0000", 2024-04-26T01:09:01.342 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4839999999999998e-06 2024-04-26T01:09:01.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.343 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.343 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.343 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361892+0000", 2024-04-26T01:09:01.343 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.437e-06 2024-04-26T01:09:01.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.343 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.343 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.344 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.363374+0000", 2024-04-26T01:09:01.344 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001482693 2024-04-26T01:09:01.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.344 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.344 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:01.344 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.363442+0000", 2024-04-26T01:09:01.344 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7448999999999997e-05 2024-04-26T01:09:01.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.345 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.345 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.345 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.363805+0000", 2024-04-26T01:09:01.345 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00036289900000000002 2024-04-26T01:09:01.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.345 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.345 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.345 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.366199+0000", 2024-04-26T01:09:01.345 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.002394654 2024-04-26T01:09:01.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.345 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.345 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:09:01.346 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472968+0000", 2024-04-26T01:09:01.346 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.106769087 2024-04-26T01:09:01.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.346 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.346 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.346 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473060+0000", 2024-04-26T01:09:01.346 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.1218000000000007e-05 2024-04-26T01:09:01.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.346 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.346 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:63 3.es0 3:7eef64f7:::smithi16428812-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.361913+0000", 2024-04-26T01:09:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 61.958356887000001, 2024-04-26T01:09:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.11409399200000001, 2024-04-26T01:09:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:09:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 63 2024-04-26T01:09:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361913+0000", 2024-04-26T01:09:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361913+0000", 2024-04-26T01:09:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361914+0000", 2024-04-26T01:09:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.5700000000000002e-07 2024-04-26T01:09:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361917+0000", 2024-04-26T01:09:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0979999999999998e-06 2024-04-26T01:09:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361923+0000", 2024-04-26T01:09:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9719999999999997e-06 2024-04-26T01:09:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.361927+0000", 2024-04-26T01:09:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.5070000000000002e-06 2024-04-26T01:09:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.364059+0000", 2024-04-26T01:09:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.002132278 2024-04-26T01:09:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.364206+0000", 2024-04-26T01:09:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014703700000000001 2024-04-26T01:09:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473123+0000", 2024-04-26T01:09:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.108916164 2024-04-26T01:09:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473211+0000", 2024-04-26T01:09:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8381999999999995e-05 2024-04-26T01:09:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.476007+0000", 2024-04-26T01:09:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0027959970000000002 2024-04-26T01:09:01.352 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.352 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.352 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.352 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWriteReply(3.es0 33/30 ECSubWriteReply(tid=5, last_complete=33'4, committed=1, applied=1))", 2024-04-26T01:09:01.352 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.365083+0000", 2024-04-26T01:09:01.352 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 61.955186404999999, 2024-04-26T01:09:01.352 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.107988984, 2024-04-26T01:09:01.352 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.352 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.352 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.353 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.353 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.353 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365083+0000", 2024-04-26T01:09:01.353 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.353 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.353 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365083+0000", 2024-04-26T01:09:01.353 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.353 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.353 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365084+0000", 2024-04-26T01:09:01.353 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.9299999999999999e-07 2024-04-26T01:09:01.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.354 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.354 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.365086+0000", 2024-04-26T01:09:01.354 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5239999999999999e-06 2024-04-26T01:09:01.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.354 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.354 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472672+0000", 2024-04-26T01:09:01.354 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.10758553899999999 2024-04-26T01:09:01.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.355 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.355 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.355 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472680+0000", 2024-04-26T01:09:01.355 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.6820000000000003e-06 2024-04-26T01:09:01.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.355 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.355 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.355 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.472900+0000", 2024-04-26T01:09:01.355 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00022041899999999999 2024-04-26T01:09:01.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.355 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.355 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.356 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.473072+0000", 2024-04-26T01:09:01.356 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017242700000000001 2024-04-26T01:09:01.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.356 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.356 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.356 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.2s1 33/30 ECSubWrite(tid=165, reqid=client.4521.0:80, at_version=33'6, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:09:01.356 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.391943+0000", 2024-04-26T01:09:01.356 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 61.928326726999998, 2024-04-26T01:09:01.356 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.042848432999999998, 2024-04-26T01:09:01.356 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.356 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.356 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.357 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.357 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.357 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.357 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391943+0000", 2024-04-26T01:09:01.357 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.357 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.357 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.357 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391943+0000", 2024-04-26T01:09:01.357 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.357 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.357 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.357 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391944+0000", 2024-04-26T01:09:01.358 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8199999999999998e-07 2024-04-26T01:09:01.358 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.358 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.358 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.358 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.391961+0000", 2024-04-26T01:09:01.358 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6546999999999999e-05 2024-04-26T01:09:01.358 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.358 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.358 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.358 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.430981+0000", 2024-04-26T01:09:01.359 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.039020817999999999 2024-04-26T01:09:01.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.359 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.359 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.359 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.430991+0000", 2024-04-26T01:09:01.359 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7059999999999999e-06 2024-04-26T01:09:01.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.359 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.359 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.359 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.431059+0000", 2024-04-26T01:09:01.359 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.8198e-05 2024-04-26T01:09:01.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.359 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.359 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.360 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.431067+0000", 2024-04-26T01:09:01.360 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.8930000000000005e-06 2024-04-26T01:09:01.360 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.360 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.360 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.360 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.434752+0000", 2024-04-26T01:09:01.360 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0036849249999999999 2024-04-26T01:09:01.360 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.360 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.360 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.360 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.434792+0000", 2024-04-26T01:09:01.360 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.9464e-05 2024-04-26T01:09:01.360 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.360 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.360 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.361 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.361 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.361 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.3s4 33/30 ECSubWrite(tid=170, reqid=client.4521.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=33'2))", 2024-04-26T01:09:01.361 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.441271+0000", 2024-04-26T01:09:01.361 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 61.878998949, 2024-04-26T01:09:01.361 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.063989213000000003, 2024-04-26T01:09:01.361 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.361 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.361 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.361 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.361 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.361 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.361 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441271+0000", 2024-04-26T01:09:01.361 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.362 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.362 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.362 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441271+0000", 2024-04-26T01:09:01.362 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.362 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.362 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.362 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441272+0000", 2024-04-26T01:09:01.362 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1650000000000001e-06 2024-04-26T01:09:01.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.362 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.362 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.363 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.441286+0000", 2024-04-26T01:09:01.363 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3771000000000001e-05 2024-04-26T01:09:01.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.363 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.363 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.363 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504639+0000", 2024-04-26T01:09:01.363 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.063352787999999993 2024-04-26T01:09:01.364 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.364 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.364 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.364 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504647+0000", 2024-04-26T01:09:01.364 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9389999999999997e-06 2024-04-26T01:09:01.364 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.364 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.364 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.365 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504715+0000", 2024-04-26T01:09:01.365 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7926e-05 2024-04-26T01:09:01.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.365 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.365 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.365 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504722+0000", 2024-04-26T01:09:01.366 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9449999999999993e-06 2024-04-26T01:09:01.366 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.366 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.366 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.366 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.505203+0000", 2024-04-26T01:09:01.366 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00048092399999999998 2024-04-26T01:09:01.367 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.367 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.367 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.367 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.505260+0000", 2024-04-26T01:09:01.367 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.6755000000000001e-05 2024-04-26T01:09:01.367 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.367 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.368 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.368 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.368 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.368 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:233 3.es0 3:7b1f3e32:::smithi16428812-8:head [copy-from ver 4] snapc 3=[2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-26T01:09:01.368 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:02.464122+0000", 2024-04-26T01:09:01.368 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 58.856148306999998, 2024-04-26T01:09:01.368 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.66639822699999995, 2024-04-26T01:09:01.368 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.369 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.369 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:01.369 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:01.369 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:01.369 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:01.369 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 233 2024-04-26T01:09:01.369 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.369 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.370 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.370 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.370 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464122+0000", 2024-04-26T01:09:01.370 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.370 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.370 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.371 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.371 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464122+0000", 2024-04-26T01:09:01.371 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.371 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.371 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.371 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.371 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464125+0000", 2024-04-26T01:09:01.372 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0939999999999999e-06 2024-04-26T01:09:01.372 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.372 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.372 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.372 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464136+0000", 2024-04-26T01:09:01.372 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1790000000000001e-05 2024-04-26T01:09:01.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.373 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.373 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.373 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464160+0000", 2024-04-26T01:09:01.373 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3683e-05 2024-04-26T01:09:01.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.373 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.373 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.373 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464174+0000", 2024-04-26T01:09:01.373 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.382e-05 2024-04-26T01:09:01.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.373 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.373 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.373 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464324+0000", 2024-04-26T01:09:01.374 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014991900000000001 2024-04-26T01:09:01.374 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.374 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.374 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:01.374 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464485+0000", 2024-04-26T01:09:01.374 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016099500000000001 2024-04-26T01:09:01.374 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.374 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.374 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.374 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480641+0000", 2024-04-26T01:09:01.374 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016155748000000001 2024-04-26T01:09:01.374 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.374 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.374 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.375 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.485807+0000", 2024-04-26T01:09:01.375 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0051663380000000004 2024-04-26T01:09:01.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.375 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.375 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:09:01.375 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.130287+0000", 2024-04-26T01:09:01.375 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.64448033500000002 2024-04-26T01:09:01.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.375 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.375 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.375 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.130520+0000", 2024-04-26T01:09:01.375 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00023250500000000001 2024-04-26T01:09:01.375 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.375 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.375 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.376 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.376 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.376 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:234 3.es0 3:7b1f3e32:::smithi16428812-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-04-26T01:09:01.376 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:02.464220+0000", 2024-04-26T01:09:01.376 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 58.856049964, 2024-04-26T01:09:01.376 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.019428925, 2024-04-26T01:09:01.376 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.376 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.376 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:09:01.376 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:01.376 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:01.376 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:01.376 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 234 2024-04-26T01:09:01.376 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.376 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.377 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.377 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.377 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464220+0000", 2024-04-26T01:09:01.377 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.377 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.377 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.377 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464220+0000", 2024-04-26T01:09:01.377 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.377 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.377 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.377 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464221+0000", 2024-04-26T01:09:01.377 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4500000000000001e-06 2024-04-26T01:09:01.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.378 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.378 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.378 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464229+0000", 2024-04-26T01:09:01.378 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.6229999999999997e-06 2024-04-26T01:09:01.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.378 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.378 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.378 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464250+0000", 2024-04-26T01:09:01.378 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1163e-05 2024-04-26T01:09:01.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.378 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.378 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.378 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464267+0000", 2024-04-26T01:09:01.378 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.668e-05 2024-04-26T01:09:01.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.379 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.379 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.379 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464815+0000", 2024-04-26T01:09:01.379 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00054803000000000002 2024-04-26T01:09:01.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.379 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.380 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for blocked object", 2024-04-26T01:09:01.380 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.464939+0000", 2024-04-26T01:09:01.380 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000124385 2024-04-26T01:09:01.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.380 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.380 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.380 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.483264+0000", 2024-04-26T01:09:01.381 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.018324427000000001 2024-04-26T01:09:01.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.381 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.381 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:01.381 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.483429+0000", 2024-04-26T01:09:01.381 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000165749 2024-04-26T01:09:01.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.381 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.382 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.382 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.483649+0000", 2024-04-26T01:09:01.382 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00021941800000000001 2024-04-26T01:09:01.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.382 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.382 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.383 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:257 3.es0 3:7b1f3e32:::smithi16428812-8:head [append 0~0] snapc 4=[4,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:09:01.383 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:03.446919+0000", 2024-04-26T01:09:01.383 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 57.873350997999999, 2024-04-26T01:09:01.383 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.84360518900000003, 2024-04-26T01:09:01.383 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.383 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.383 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:01.383 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:01.384 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:01.384 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:01.384 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 257 2024-04-26T01:09:01.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.384 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.384 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.384 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.384 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446919+0000", 2024-04-26T01:09:01.384 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.385 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.385 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.385 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446919+0000", 2024-04-26T01:09:01.385 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.385 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.385 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.386 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446920+0000", 2024-04-26T01:09:01.386 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4700000000000001e-07 2024-04-26T01:09:01.386 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.386 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.386 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.386 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446922+0000", 2024-04-26T01:09:01.386 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6800000000000002e-06 2024-04-26T01:09:01.386 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.387 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.387 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.387 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446931+0000", 2024-04-26T01:09:01.387 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8659999999999995e-06 2024-04-26T01:09:01.387 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.387 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.387 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.387 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.446941+0000", 2024-04-26T01:09:01.387 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.5540000000000003e-06 2024-04-26T01:09:01.388 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.388 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.388 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.388 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.458756+0000", 2024-04-26T01:09:01.388 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011815275 2024-04-26T01:09:01.388 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.388 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.388 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:01.388 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.459005+0000", 2024-04-26T01:09:01.389 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000249418 2024-04-26T01:09:01.389 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.389 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.389 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.389 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.460129+0000", 2024-04-26T01:09:01.389 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001123768 2024-04-26T01:09:01.389 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.389 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.390 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.390 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461504+0000", 2024-04-26T01:09:01.390 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001375079 2024-04-26T01:09:01.390 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.390 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.390 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:09:01.390 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.290324+0000", 2024-04-26T01:09:01.390 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.82881998700000004 2024-04-26T01:09:01.390 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.391 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.391 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.391 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.290524+0000", 2024-04-26T01:09:01.391 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000199815 2024-04-26T01:09:01.391 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.391 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.391 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.392 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.392 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.392 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/30 ECSubWriteReply(tid=25, last_complete=37'19, committed=1, applied=1))", 2024-04-26T01:09:01.392 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:03.461054+0000", 2024-04-26T01:09:01.392 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 57.859216009999997, 2024-04-26T01:09:01.392 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.83058926899999996, 2024-04-26T01:09:01.392 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.392 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.392 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.393 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.393 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.393 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.393 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461054+0000", 2024-04-26T01:09:01.393 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.393 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.393 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.393 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461054+0000", 2024-04-26T01:09:01.394 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.394 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.394 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.394 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461056+0000", 2024-04-26T01:09:01.394 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.906e-06 2024-04-26T01:09:01.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.394 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.461069+0000", 2024-04-26T01:09:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.295e-05 2024-04-26T01:09:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.289919+0000", 2024-04-26T01:09:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.82884987300000001 2024-04-26T01:09:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.396 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.396 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.396 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.289933+0000", 2024-04-26T01:09:01.396 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4430000000000001e-05 2024-04-26T01:09:01.396 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.396 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.396 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.396 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.290082+0000", 2024-04-26T01:09:01.396 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014943899999999999 2024-04-26T01:09:01.396 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.396 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.396 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.396 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.291643+0000", 2024-04-26T01:09:01.396 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001560671 2024-04-26T01:09:01.396 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.397 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.397 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.397 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.397 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:385 3.fs1 3.db40cadf (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:09:01.397 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.452224+0000", 2024-04-26T01:09:01.397 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 51.868046259000003, 2024-04-26T01:09:01.397 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95265389199999995, 2024-04-26T01:09:01.397 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.397 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.397 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.397 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:01.397 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:01.397 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:01.397 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 385 2024-04-26T01:09:01.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.398 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.398 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.398 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.398 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452224+0000", 2024-04-26T01:09:01.398 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.398 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.398 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.398 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452224+0000", 2024-04-26T01:09:01.398 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.398 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.398 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.399 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452227+0000", 2024-04-26T01:09:01.399 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6830000000000001e-06 2024-04-26T01:09:01.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.399 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.399 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.399 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452242+0000", 2024-04-26T01:09:01.399 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4493e-05 2024-04-26T01:09:01.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.399 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.399 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.399 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452274+0000", 2024-04-26T01:09:01.399 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1806e-05 2024-04-26T01:09:01.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.399 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.399 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.400 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452300+0000", 2024-04-26T01:09:01.400 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6092999999999999e-05 2024-04-26T01:09:01.400 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.400 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.400 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.400 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452469+0000", 2024-04-26T01:09:01.400 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000169629 2024-04-26T01:09:01.400 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.400 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.400 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:09:01.400 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452475+0000", 2024-04-26T01:09:01.400 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.604e-06 2024-04-26T01:09:01.400 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.400 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.400 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.401 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.404853+0000", 2024-04-26T01:09:01.401 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95237828400000002 2024-04-26T01:09:01.401 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.401 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.401 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.401 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.404878+0000", 2024-04-26T01:09:01.401 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4300000000000001e-05 2024-04-26T01:09:01.401 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.401 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.401 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.401 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.401 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.401 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:386 3.fs1 3.db40cadf (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:09:01.402 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.452348+0000", 2024-04-26T01:09:01.402 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 51.867922057000001, 2024-04-26T01:09:01.402 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95267636899999997, 2024-04-26T01:09:01.402 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.402 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.402 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.402 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:01.402 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:01.402 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:01.402 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 386 2024-04-26T01:09:01.402 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.403 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.403 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.403 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.403 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452348+0000", 2024-04-26T01:09:01.403 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.403 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.403 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.403 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452348+0000", 2024-04-26T01:09:01.403 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.403 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.403 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.403 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452349+0000", 2024-04-26T01:09:01.403 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.533e-06 2024-04-26T01:09:01.404 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.404 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.404 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.404 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452359+0000", 2024-04-26T01:09:01.404 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.5070000000000006e-06 2024-04-26T01:09:01.404 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.404 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.404 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.404 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452381+0000", 2024-04-26T01:09:01.404 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2306e-05 2024-04-26T01:09:01.404 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.404 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.404 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.404 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452398+0000", 2024-04-26T01:09:01.404 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6456e-05 2024-04-26T01:09:01.405 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.405 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.405 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.405 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452605+0000", 2024-04-26T01:09:01.405 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00020750900000000001 2024-04-26T01:09:01.405 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.405 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.405 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:09:01.405 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452610+0000", 2024-04-26T01:09:01.405 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3569999999999997e-06 2024-04-26T01:09:01.405 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.405 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.405 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.405 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.404988+0000", 2024-04-26T01:09:01.406 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95237776100000004 2024-04-26T01:09:01.406 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.406 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.406 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.406 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405024+0000", 2024-04-26T01:09:01.406 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5939999999999998e-05 2024-04-26T01:09:01.406 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.406 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.406 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.406 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.406 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.406 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:387 3.fs1 3.db40cadf (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:09:01.406 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.452423+0000", 2024-04-26T01:09:01.406 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 51.867846651000001, 2024-04-26T01:09:01.407 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.95294480000000004, 2024-04-26T01:09:01.407 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.407 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.407 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.407 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:01.407 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:01.407 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:01.407 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 387 2024-04-26T01:09:01.407 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.407 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.407 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.407 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.407 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452423+0000", 2024-04-26T01:09:01.407 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.408 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.408 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.408 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.408 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452423+0000", 2024-04-26T01:09:01.408 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.408 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.408 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.408 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.408 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452425+0000", 2024-04-26T01:09:01.408 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.373e-06 2024-04-26T01:09:01.408 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.408 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.408 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.408 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452435+0000", 2024-04-26T01:09:01.408 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0149999999999999e-05 2024-04-26T01:09:01.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.409 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.409 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.409 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452447+0000", 2024-04-26T01:09:01.409 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.199e-05 2024-04-26T01:09:01.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.409 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.409 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.409 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452461+0000", 2024-04-26T01:09:01.409 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4503e-05 2024-04-26T01:09:01.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.409 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.409 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.409 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452685+0000", 2024-04-26T01:09:01.410 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00022367200000000001 2024-04-26T01:09:01.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.410 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:09:01.410 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.452687+0000", 2024-04-26T01:09:01.410 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.446e-06 2024-04-26T01:09:01.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.410 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.410 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405319+0000", 2024-04-26T01:09:01.410 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.952631279 2024-04-26T01:09:01.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.410 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.411 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405368+0000", 2024-04-26T01:09:01.411 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9387e-05 2024-04-26T01:09:01.411 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.411 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.411 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.411 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.411 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:393 3.fs1 3.94f9353f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:09:01.411 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.464778+0000", 2024-04-26T01:09:01.411 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 51.855492122999998, 2024-04-26T01:09:01.411 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94080343600000005, 2024-04-26T01:09:01.411 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.411 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.411 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.412 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:01.412 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:01.412 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:01.412 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 393 2024-04-26T01:09:01.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.412 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.412 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.412 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.412 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464778+0000", 2024-04-26T01:09:01.412 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.413 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.413 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.413 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464778+0000", 2024-04-26T01:09:01.413 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.413 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.413 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.413 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464780+0000", 2024-04-26T01:09:01.413 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2460000000000002e-06 2024-04-26T01:09:01.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.413 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.413 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.413 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464791+0000", 2024-04-26T01:09:01.414 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.145e-05 2024-04-26T01:09:01.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.414 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.414 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.414 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464823+0000", 2024-04-26T01:09:01.414 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1157e-05 2024-04-26T01:09:01.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.414 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.414 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.414 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464837+0000", 2024-04-26T01:09:01.414 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.488e-05 2024-04-26T01:09:01.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.414 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.414 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.415 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464985+0000", 2024-04-26T01:09:01.415 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000147999 2024-04-26T01:09:01.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.415 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:09:01.415 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464991+0000", 2024-04-26T01:09:01.415 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3329999999999996e-06 2024-04-26T01:09:01.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.415 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.415 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405522+0000", 2024-04-26T01:09:01.415 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94053098899999998 2024-04-26T01:09:01.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.415 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.416 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.405581+0000", 2024-04-26T01:09:01.416 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9382000000000002e-05 2024-04-26T01:09:01.416 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.416 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.416 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.416 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:394 3.fs1 3.94f9353f (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-26T01:09:01.416 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.464870+0000", 2024-04-26T01:09:01.416 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 51.855399847000001, 2024-04-26T01:09:01.416 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94122545000000002, 2024-04-26T01:09:01.416 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.416 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.417 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.417 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:01.417 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:01.417 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:01.417 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 394 2024-04-26T01:09:01.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.417 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.417 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.417 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464870+0000", 2024-04-26T01:09:01.417 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.418 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.418 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464870+0000", 2024-04-26T01:09:01.418 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.418 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.418 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464871+0000", 2024-04-26T01:09:01.418 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1230000000000001e-06 2024-04-26T01:09:01.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.418 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.418 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464875+0000", 2024-04-26T01:09:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.25e-06 2024-04-26T01:09:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464895+0000", 2024-04-26T01:09:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9332999999999999e-05 2024-04-26T01:09:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.464904+0000", 2024-04-26T01:09:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9230000000000008e-06 2024-04-26T01:09:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.465051+0000", 2024-04-26T01:09:01.420 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000147186 2024-04-26T01:09:01.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.420 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for peered", 2024-04-26T01:09:01.420 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.465054+0000", 2024-04-26T01:09:01.420 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7530000000000002e-06 2024-04-26T01:09:01.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.420 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.420 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.406060+0000", 2024-04-26T01:09:01.420 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.94100597900000005 2024-04-26T01:09:01.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.420 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.420 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:10.406095+0000", 2024-04-26T01:09:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5902999999999998e-05 2024-04-26T01:09:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpRead(3.fs1 58/44 ECSubRead(tid=311, to_read={3:fa21e776:::smithi16428812-18:head=0,4096,0}, subchunks={3:fa21e776:::smithi16428812-18:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:09:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:24.214139+0000", 2024-04-26T01:09:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 37.106130706000002, 2024-04-26T01:09:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.068278328999999999, 2024-04-26T01:09:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.422 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.422 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.214139+0000", 2024-04-26T01:09:01.422 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.422 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.422 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.422 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.214139+0000", 2024-04-26T01:09:01.422 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.422 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.423 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.423 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.214141+0000", 2024-04-26T01:09:01.423 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.674e-06 2024-04-26T01:09:01.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.423 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.423 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.423 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.214161+0000", 2024-04-26T01:09:01.423 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0349000000000001e-05 2024-04-26T01:09:01.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.423 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.423 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.423 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.281949+0000", 2024-04-26T01:09:01.423 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.067788082999999999 2024-04-26T01:09:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.281978+0000", 2024-04-26T01:09:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8646999999999999e-05 2024-04-26T01:09:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.282180+0000", 2024-04-26T01:09:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00020251500000000001 2024-04-26T01:09:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.282417+0000", 2024-04-26T01:09:01.425 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000237061 2024-04-26T01:09:01.425 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.425 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.425 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.425 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:692 3.es0 3:7eef64f7:::smithi16428812-21:head [read 0~2719744,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-26T01:09:01.425 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:30.194619+0000", 2024-04-26T01:09:01.425 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 31.125650411999999, 2024-04-26T01:09:01.425 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.021611364000000001, 2024-04-26T01:09:01.425 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.425 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.426 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:09:01.426 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:01.426 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:01.426 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:01.426 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 692 2024-04-26T01:09:01.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.426 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.426 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.426 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.426 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194619+0000", 2024-04-26T01:09:01.427 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.427 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.427 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194619+0000", 2024-04-26T01:09:01.427 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.427 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.427 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194620+0000", 2024-04-26T01:09:01.427 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7300000000000004e-07 2024-04-26T01:09:01.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.427 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.427 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194641+0000", 2024-04-26T01:09:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0910000000000001e-05 2024-04-26T01:09:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194653+0000", 2024-04-26T01:09:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1663e-05 2024-04-26T01:09:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194661+0000", 2024-04-26T01:09:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.49e-06 2024-04-26T01:09:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194940+0000", 2024-04-26T01:09:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00027803799999999998 2024-04-26T01:09:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:09:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.194956+0000", 2024-04-26T01:09:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6702999999999999e-05 2024-04-26T01:09:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.201697+0000", 2024-04-26T01:09:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0067405629999999998 2024-04-26T01:09:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.201779+0000", 2024-04-26T01:09:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2678999999999999e-05 2024-04-26T01:09:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:30.216231+0000", 2024-04-26T01:09:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014451345000000001 2024-04-26T01:09:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:902 3.es0 3:7b1f3e32:::smithi16428812-8:head [copy-from ver 32] snapc 1b=[1b,1a,19,18] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:09:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:47.482197+0000", 2024-04-26T01:09:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.838072621, 2024-04-26T01:09:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.89556173800000005, 2024-04-26T01:09:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 902 2024-04-26T01:09:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482197+0000", 2024-04-26T01:09:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.433 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.433 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482197+0000", 2024-04-26T01:09:01.433 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.433 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.433 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482199+0000", 2024-04-26T01:09:01.433 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5710000000000001e-06 2024-04-26T01:09:01.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.434 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.434 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482205+0000", 2024-04-26T01:09:01.434 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.489e-06 2024-04-26T01:09:01.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.434 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.434 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.434 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482218+0000", 2024-04-26T01:09:01.434 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2996000000000001e-05 2024-04-26T01:09:01.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.434 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.434 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.434 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482226+0000", 2024-04-26T01:09:01.434 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5510000000000002e-06 2024-04-26T01:09:01.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482371+0000", 2024-04-26T01:09:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014525599999999999 2024-04-26T01:09:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482652+0000", 2024-04-26T01:09:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000281165 2024-04-26T01:09:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.367066+0000", 2024-04-26T01:09:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.88441396500000002 2024-04-26T01:09:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.436 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.436 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.436 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.375685+0000", 2024-04-26T01:09:01.436 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0086183449999999995 2024-04-26T01:09:01.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.436 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.436 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:09:01.436 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.377540+0000", 2024-04-26T01:09:01.436 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001855659 2024-04-26T01:09:01.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.436 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.436 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.437 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.377759+0000", 2024-04-26T01:09:01.437 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000218741 2024-04-26T01:09:01.437 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.437 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.437 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.437 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.437 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.437 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:903 3.es0 3:7b1f3e32:::smithi16428812-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:09:01.437 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:47.482246+0000", 2024-04-26T01:09:01.437 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.838023403999999, 2024-04-26T01:09:01.437 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.89304915500000004, 2024-04-26T01:09:01.437 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.437 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.437 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:09:01.437 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:01.438 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:01.438 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:01.438 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 903 2024-04-26T01:09:01.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.438 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.438 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.438 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.438 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482246+0000", 2024-04-26T01:09:01.438 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.438 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482246+0000", 2024-04-26T01:09:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482247+0000", 2024-04-26T01:09:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7500000000000001e-07 2024-04-26T01:09:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482249+0000", 2024-04-26T01:09:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8840000000000001e-06 2024-04-26T01:09:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.440 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.440 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482253+0000", 2024-04-26T01:09:01.440 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.4440000000000001e-06 2024-04-26T01:09:01.440 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.440 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.440 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.440 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.482256+0000", 2024-04-26T01:09:01.440 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1700000000000001e-06 2024-04-26T01:09:01.440 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.440 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.440 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.441 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.483084+0000", 2024-04-26T01:09:01.441 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00082729500000000003 2024-04-26T01:09:01.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.441 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.441 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for blocked object", 2024-04-26T01:09:01.441 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.483147+0000", 2024-04-26T01:09:01.441 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.3812999999999994e-05 2024-04-26T01:09:01.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.441 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.441 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.442 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.374930+0000", 2024-04-26T01:09:01.442 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.89178298199999995 2024-04-26T01:09:01.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.442 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:01.442 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.375134+0000", 2024-04-26T01:09:01.442 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00020325599999999999 2024-04-26T01:09:01.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.442 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.442 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.375296+0000", 2024-04-26T01:09:01.442 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000161936 2024-04-26T01:09:01.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.443 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.443 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.443 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:09:01.444 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:09:01.406+0000 7fd70ac967c0 -1 osd.3 79 log_to_monitors true 2024-04-26T01:09:01.445 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:09:01.575 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:09:01.575 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:09:01.576 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:09:01.576 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:09:01.589 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:09:01.711 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:09:01.711 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:09:01.711 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:09:01.711 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:09:01.711 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:09:01.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:09:01.845 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:09:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:09:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:09:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:09:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 36/30 ECSubWrite(tid=15, reqid=client.4521.0:233, at_version=36'11, trim_to=0'0, roll_forward_to=33'9))", 2024-04-26T01:09:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:02.480147+0000", 2024-04-26T01:09:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 59.364141603999997, 2024-04-26T01:09:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.64949222799999995, 2024-04-26T01:09:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480147+0000", 2024-04-26T01:09:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480147+0000", 2024-04-26T01:09:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480148+0000", 2024-04-26T01:09:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2300000000000001e-06 2024-04-26T01:09:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:02.480517+0000", 2024-04-26T01:09:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00036900700000000003 2024-04-26T01:09:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125262+0000", 2024-04-26T01:09:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.64474460600000005 2024-04-26T01:09:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125278+0000", 2024-04-26T01:09:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5622999999999999e-05 2024-04-26T01:09:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125483+0000", 2024-04-26T01:09:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000205105 2024-04-26T01:09:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.125502+0000", 2024-04-26T01:09:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8876999999999999e-05 2024-04-26T01:09:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.129565+0000", 2024-04-26T01:09:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0040633930000000002 2024-04-26T01:09:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:03.129639+0000", 2024-04-26T01:09:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4387000000000005e-05 2024-04-26T01:09:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.5s6 43/30 ECSubWrite(tid=35, reqid=osd.15.0:35, at_version=43'30, trim_to=0'0, roll_forward_to=42'28))", 2024-04-26T01:09:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:09:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 52.437729863999998, 2024-04-26T01:09:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011310863000000001, 2024-04-26T01:09:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:09:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:09:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406559+0000", 2024-04-26T01:09:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2e-07 2024-04-26T01:09:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406612+0000", 2024-04-26T01:09:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2206000000000001e-05 2024-04-26T01:09:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406709+0000", 2024-04-26T01:09:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7106000000000002e-05 2024-04-26T01:09:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.406737+0000", 2024-04-26T01:09:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8629999999999999e-05 2024-04-26T01:09:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.407351+0000", 2024-04-26T01:09:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00061410800000000004 2024-04-26T01:09:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.407377+0000", 2024-04-26T01:09:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5057000000000001e-05 2024-04-26T01:09:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.417771+0000", 2024-04-26T01:09:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.01039402 2024-04-26T01:09:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:09.417870+0000", 2024-04-26T01:09:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9215999999999993e-05 2024-04-26T01:09:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 50/44 ECSubWrite(tid=51, reqid=osd.2.0:51, at_version=50'40, trim_to=0'0, roll_forward_to=45'38))", 2024-04-26T01:09:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.058137+0000", 2024-04-26T01:09:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 45.786151482000001, 2024-04-26T01:09:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0092019839999999999, 2024-04-26T01:09:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058137+0000", 2024-04-26T01:09:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058137+0000", 2024-04-26T01:09:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058138+0000", 2024-04-26T01:09:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.3300000000000001e-07 2024-04-26T01:09:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058197+0000", 2024-04-26T01:09:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9110000000000002e-05 2024-04-26T01:09:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058307+0000", 2024-04-26T01:09:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000109706 2024-04-26T01:09:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058330+0000", 2024-04-26T01:09:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3540000000000002e-05 2024-04-26T01:09:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058521+0000", 2024-04-26T01:09:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019083500000000001 2024-04-26T01:09:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.058548+0000", 2024-04-26T01:09:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6950000000000001e-05 2024-04-26T01:09:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.067237+0000", 2024-04-26T01:09:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0086888769999999994 2024-04-26T01:09:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.067339+0000", 2024-04-26T01:09:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010213300000000001 2024-04-26T01:09:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs4 50/44 ECSubWrite(tid=22, reqid=osd.9.0:22, at_version=50'37, trim_to=0'0, roll_forward_to=47'35))", 2024-04-26T01:09:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:09:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 45.785080163000003, 2024-04-26T01:09:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016916468, 2024-04-26T01:09:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:09:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:09:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059209+0000", 2024-04-26T01:09:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.9299999999999996e-07 2024-04-26T01:09:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059282+0000", 2024-04-26T01:09:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2019000000000003e-05 2024-04-26T01:09:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059397+0000", 2024-04-26T01:09:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000115569 2024-04-26T01:09:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059419+0000", 2024-04-26T01:09:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1497000000000001e-05 2024-04-26T01:09:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064015+0000", 2024-04-26T01:09:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0045966059999999996 2024-04-26T01:09:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064041+0000", 2024-04-26T01:09:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5570000000000001e-05 2024-04-26T01:09:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.075972+0000", 2024-04-26T01:09:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011930802000000001 2024-04-26T01:09:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076125+0000", 2024-04-26T01:09:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015351199999999999 2024-04-26T01:09:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 50/44 ECSubWrite(tid=215, reqid=client.4521.0:490, at_version=50'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-26T01:09:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.059258+0000", 2024-04-26T01:09:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 45.785030290999998, 2024-04-26T01:09:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014073085000000001, 2024-04-26T01:09:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059258+0000", 2024-04-26T01:09:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059258+0000", 2024-04-26T01:09:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059261+0000", 2024-04-26T01:09:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3829999999999999e-06 2024-04-26T01:09:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.059926+0000", 2024-04-26T01:09:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00066483199999999997 2024-04-26T01:09:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060011+0000", 2024-04-26T01:09:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5248999999999996e-05 2024-04-26T01:09:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060033+0000", 2024-04-26T01:09:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1566999999999999e-05 2024-04-26T01:09:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060387+0000", 2024-04-26T01:09:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000354793 2024-04-26T01:09:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060449+0000", 2024-04-26T01:09:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.1957000000000003e-05 2024-04-26T01:09:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073174+0000", 2024-04-26T01:09:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012724998 2024-04-26T01:09:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073332+0000", 2024-04-26T01:09:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015730599999999999 2024-04-26T01:09:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.0s5 50/30 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=50'32, trim_to=0'0, roll_forward_to=45'30))", 2024-04-26T01:09:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.062205+0000", 2024-04-26T01:09:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 45.782083911000001, 2024-04-26T01:09:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.01120789, 2024-04-26T01:09:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062205+0000", 2024-04-26T01:09:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062205+0000", 2024-04-26T01:09:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062207+0000", 2024-04-26T01:09:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.844e-06 2024-04-26T01:09:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062258+0000", 2024-04-26T01:09:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.1388999999999999e-05 2024-04-26T01:09:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062376+0000", 2024-04-26T01:09:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00011767199999999999 2024-04-26T01:09:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062399+0000", 2024-04-26T01:09:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3410000000000001e-05 2024-04-26T01:09:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062583+0000", 2024-04-26T01:09:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00018336900000000001 2024-04-26T01:09:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.062611+0000", 2024-04-26T01:09:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8189999999999999e-05 2024-04-26T01:09:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073316+0000", 2024-04-26T01:09:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010704824 2024-04-26T01:09:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073413+0000", 2024-04-26T01:09:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7192000000000001e-05 2024-04-26T01:09:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.2s3 50/44 ECSubWrite(tid=212, reqid=osd.13.0:212, at_version=50'24, trim_to=0'0, roll_forward_to=45'22))", 2024-04-26T01:09:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.063661+0000", 2024-04-26T01:09:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 45.780627293000002, 2024-04-26T01:09:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012570563999999999, 2024-04-26T01:09:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063661+0000", 2024-04-26T01:09:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063661+0000", 2024-04-26T01:09:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063662+0000", 2024-04-26T01:09:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.37e-07 2024-04-26T01:09:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063721+0000", 2024-04-26T01:09:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.8696999999999997e-05 2024-04-26T01:09:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063860+0000", 2024-04-26T01:09:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013883799999999999 2024-04-26T01:09:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063882+0000", 2024-04-26T01:09:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2313000000000001e-05 2024-04-26T01:09:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064661+0000", 2024-04-26T01:09:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00077955699999999995 2024-04-26T01:09:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.064686+0000", 2024-04-26T01:09:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4984e-05 2024-04-26T01:09:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.075967+0000", 2024-04-26T01:09:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011280646 2024-04-26T01:09:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076232+0000", 2024-04-26T01:09:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00026499200000000003 2024-04-26T01:09:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 50/30 ECSubWrite(tid=216, reqid=osd.4.0:214, at_version=0'0, trim_to=0'0, roll_forward_to=50'22))", 2024-04-26T01:09:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.065107+0000", 2024-04-26T01:09:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 45.779181391000002, 2024-04-26T01:09:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0087328609999999997, 2024-04-26T01:09:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065107+0000", 2024-04-26T01:09:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065107+0000", 2024-04-26T01:09:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065108+0000", 2024-04-26T01:09:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.0999999999999997e-07 2024-04-26T01:09:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065125+0000", 2024-04-26T01:09:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7010000000000001e-05 2024-04-26T01:09:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065360+0000", 2024-04-26T01:09:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00023507199999999999 2024-04-26T01:09:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065382+0000", 2024-04-26T01:09:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1993000000000001e-05 2024-04-26T01:09:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065672+0000", 2024-04-26T01:09:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00028928100000000001 2024-04-26T01:09:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065696+0000", 2024-04-26T01:09:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4273e-05 2024-04-26T01:09:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073748+0000", 2024-04-26T01:09:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0080520490000000004 2024-04-26T01:09:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073840+0000", 2024-04-26T01:09:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2373000000000005e-05 2024-04-26T01:09:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.7s1 50/44 ECSubWrite(tid=218, reqid=osd.4.0:218, at_version=50'34, trim_to=0'0, roll_forward_to=47'30))", 2024-04-26T01:09:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.065939+0000", 2024-04-26T01:09:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 45.778349777000003, 2024-04-26T01:09:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010065406000000001, 2024-04-26T01:09:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065939+0000", 2024-04-26T01:09:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065939+0000", 2024-04-26T01:09:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065941+0000", 2024-04-26T01:09:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.9199999999999998e-06 2024-04-26T01:09:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.065994+0000", 2024-04-26T01:09:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3439000000000001e-05 2024-04-26T01:09:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066088+0000", 2024-04-26T01:09:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.365e-05 2024-04-26T01:09:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066110+0000", 2024-04-26T01:09:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1912999999999998e-05 2024-04-26T01:09:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066779+0000", 2024-04-26T01:09:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00066918399999999997 2024-04-26T01:09:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066810+0000", 2024-04-26T01:09:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0446999999999998e-05 2024-04-26T01:09:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.075902+0000", 2024-04-26T01:09:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0090923529999999992 2024-04-26T01:09:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076004+0000", 2024-04-26T01:09:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0001025 2024-04-26T01:09:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.8s6 50/30 ECSubWrite(tid=42, reqid=osd.5.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=50'26))", 2024-04-26T01:09:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.068230+0000", 2024-04-26T01:09:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 45.776058925999997, 2024-04-26T01:09:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0080202260000000001, 2024-04-26T01:09:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068230+0000", 2024-04-26T01:09:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068230+0000", 2024-04-26T01:09:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068232+0000", 2024-04-26T01:09:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.804e-06 2024-04-26T01:09:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068261+0000", 2024-04-26T01:09:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9595999999999999e-05 2024-04-26T01:09:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068341+0000", 2024-04-26T01:09:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.9302999999999996e-05 2024-04-26T01:09:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068355+0000", 2024-04-26T01:09:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.465e-05 2024-04-26T01:09:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068495+0000", 2024-04-26T01:09:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013948500000000001 2024-04-26T01:09:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.068520+0000", 2024-04-26T01:09:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5015999999999999e-05 2024-04-26T01:09:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076177+0000", 2024-04-26T01:09:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0076572489999999997 2024-04-26T01:09:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.076250+0000", 2024-04-26T01:09:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.3122999999999998e-05 2024-04-26T01:09:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 55/30 ECSubWrite(tid=252, reqid=client.4521.0:548, at_version=55'28, trim_to=0'0, roll_forward_to=52'26))", 2024-04-26T01:09:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:21.084368+0000", 2024-04-26T01:09:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 40.759920528000002, 2024-04-26T01:09:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0068931720000000004, 2024-04-26T01:09:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084368+0000", 2024-04-26T01:09:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084368+0000", 2024-04-26T01:09:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.084369+0000", 2024-04-26T01:09:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.1999999999999999e-07 2024-04-26T01:09:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085034+0000", 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00066459099999999997 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085069+0000", 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4965999999999997e-05 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085081+0000", 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2422999999999999e-05 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085243+0000", 2024-04-26T01:09:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000162466 2024-04-26T01:09:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.085270+0000", 2024-04-26T01:09:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6775999999999999e-05 2024-04-26T01:09:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091175+0000", 2024-04-26T01:09:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059045340000000003 2024-04-26T01:09:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091261+0000", 2024-04-26T01:09:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.6695999999999998e-05 2024-04-26T01:09:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 55/44 ECSubWrite(tid=55, reqid=osd.2.0:55, at_version=55'44, trim_to=0'0, roll_forward_to=52'42))", 2024-04-26T01:09:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:21.093675+0000", 2024-04-26T01:09:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 40.750613612000002, 2024-04-26T01:09:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066519099999999996, 2024-04-26T01:09:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093675+0000", 2024-04-26T01:09:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093675+0000", 2024-04-26T01:09:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093676+0000", 2024-04-26T01:09:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.3e-07 2024-04-26T01:09:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093734+0000", 2024-04-26T01:09:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7833e-05 2024-04-26T01:09:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093836+0000", 2024-04-26T01:09:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010234900000000001 2024-04-26T01:09:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.093859+0000", 2024-04-26T01:09:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3383e-05 2024-04-26T01:09:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.094252+0000", 2024-04-26T01:09:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000392424 2024-04-26T01:09:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.094287+0000", 2024-04-26T01:09:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.5713000000000002e-05 2024-04-26T01:09:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.100231+0000", 2024-04-26T01:09:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059430519999999999 2024-04-26T01:09:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.100327+0000", 2024-04-26T01:09:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6526000000000004e-05 2024-04-26T01:09:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 59/30 ECSubWrite(tid=322, reqid=osd.4.0:322, at_version=59'45, trim_to=0'0, roll_forward_to=58'41))", 2024-04-26T01:09:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.160873+0000", 2024-04-26T01:09:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 36.683416086000001, 2024-04-26T01:09:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0064776979999999996, 2024-04-26T01:09:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160873+0000", 2024-04-26T01:09:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160873+0000", 2024-04-26T01:09:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160874+0000", 2024-04-26T01:09:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5030000000000001e-06 2024-04-26T01:09:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160924+0000", 2024-04-26T01:09:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0158999999999999e-05 2024-04-26T01:09:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161010+0000", 2024-04-26T01:09:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5285999999999996e-05 2024-04-26T01:09:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161032+0000", 2024-04-26T01:09:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2427000000000001e-05 2024-04-26T01:09:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161983+0000", 2024-04-26T01:09:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00095064900000000003 2024-04-26T01:09:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.162010+0000", 2024-04-26T01:09:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7786000000000001e-05 2024-04-26T01:09:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167263+0000", 2024-04-26T01:09:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0052521979999999996 2024-04-26T01:09:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167350+0000", 2024-04-26T01:09:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7689999999999996e-05 2024-04-26T01:09:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.5s6 72/30 ECSubWrite(tid=56, reqid=client.4521.0:785, at_version=72'49, trim_to=0'0, roll_forward_to=72'47))", 2024-04-26T01:09:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:38.301857+0000", 2024-04-26T01:09:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 23.542432016999999, 2024-04-26T01:09:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0073268409999999997, 2024-04-26T01:09:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.301857+0000", 2024-04-26T01:09:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.301857+0000", 2024-04-26T01:09:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.301859+0000", 2024-04-26T01:09:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.588e-06 2024-04-26T01:09:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.306215+0000", 2024-04-26T01:09:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0043557179999999997 2024-04-26T01:09:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.306269+0000", 2024-04-26T01:09:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.4289999999999997e-05 2024-04-26T01:09:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.306282+0000", 2024-04-26T01:09:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2332e-05 2024-04-26T01:09:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.306357+0000", 2024-04-26T01:09:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5117999999999995e-05 2024-04-26T01:09:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.306368+0000", 2024-04-26T01:09:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1063000000000001e-05 2024-04-26T01:09:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.309146+0000", 2024-04-26T01:09:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0027777040000000002 2024-04-26T01:09:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:38.309184+0000", 2024-04-26T01:09:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8028000000000003e-05 2024-04-26T01:09:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs4 82/44 ECSubWrite(tid=95, reqid=client.4521.0:914, at_version=82'96, trim_to=0'0, roll_forward_to=81'94))", 2024-04-26T01:09:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:48.332852+0000", 2024-04-26T01:09:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.511436362, 2024-04-26T01:09:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066861070000000002, 2024-04-26T01:09:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.332852+0000", 2024-04-26T01:09:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.332852+0000", 2024-04-26T01:09:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.332853+0000", 2024-04-26T01:09:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7000000000000005e-07 2024-04-26T01:09:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.332907+0000", 2024-04-26T01:09:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3706999999999999e-05 2024-04-26T01:09:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.333021+0000", 2024-04-26T01:09:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00011459900000000001 2024-04-26T01:09:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.333045+0000", 2024-04-26T01:09:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3422999999999999e-05 2024-04-26T01:09:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.333196+0000", 2024-04-26T01:09:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000151352 2024-04-26T01:09:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.333237+0000", 2024-04-26T01:09:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.1386e-05 2024-04-26T01:09:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.339469+0000", 2024-04-26T01:09:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062311399999999996 2024-04-26T01:09:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.339539+0000", 2024-04-26T01:09:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9930000000000003e-05 2024-04-26T01:09:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 82/30 ECSubWrite(tid=490, reqid=client.4521.0:912, at_version=82'73, trim_to=0'0, roll_forward_to=81'71))", 2024-04-26T01:09:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:48.355097+0000", 2024-04-26T01:09:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.489191906, 2024-04-26T01:09:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0064842040000000004, 2024-04-26T01:09:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.355097+0000", 2024-04-26T01:09:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.355097+0000", 2024-04-26T01:09:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.355097+0000", 2024-04-26T01:09:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7400000000000003e-07 2024-04-26T01:09:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.356630+0000", 2024-04-26T01:09:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0015323100000000001 2024-04-26T01:09:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.356696+0000", 2024-04-26T01:09:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.6347999999999996e-05 2024-04-26T01:09:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.356714+0000", 2024-04-26T01:09:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7762e-05 2024-04-26T01:09:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.357600+0000", 2024-04-26T01:09:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00088573999999999999 2024-04-26T01:09:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.357625+0000", 2024-04-26T01:09:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5389999999999999e-05 2024-04-26T01:09:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.361520+0000", 2024-04-26T01:09:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0038951519999999998 2024-04-26T01:09:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.361581+0000", 2024-04-26T01:09:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0927999999999997e-05 2024-04-26T01:09:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.es5 82/44 ECSubWrite(tid=81, reqid=client.4521.0:902, at_version=82'60, trim_to=0'0, roll_forward_to=81'58))", 2024-04-26T01:09:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:48.366852+0000", 2024-04-26T01:09:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.477437243000001, 2024-04-26T01:09:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0074237579999999999, 2024-04-26T01:09:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.366852+0000", 2024-04-26T01:09:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.366852+0000", 2024-04-26T01:09:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.366852+0000", 2024-04-26T01:09:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.5700000000000001e-07 2024-04-26T01:09:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.367807+0000", 2024-04-26T01:09:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00095427999999999995 2024-04-26T01:09:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.367877+0000", 2024-04-26T01:09:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.0500000000000006e-05 2024-04-26T01:09:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.367905+0000", 2024-04-26T01:09:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8119999999999998e-05 2024-04-26T01:09:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.368072+0000", 2024-04-26T01:09:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000166658 2024-04-26T01:09:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.368093+0000", 2024-04-26T01:09:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1290000000000001e-05 2024-04-26T01:09:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.374225+0000", 2024-04-26T01:09:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061319219999999997 2024-04-26T01:09:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:48.374275+0000", 2024-04-26T01:09:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0130999999999998e-05 2024-04-26T01:09:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.bs4 83/30 ECSubWrite(tid=497, reqid=osd.4.0:497, at_version=83'77, trim_to=0'0, roll_forward_to=82'73))", 2024-04-26T01:09:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:49.332826+0000", 2024-04-26T01:09:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 12.511462356999999, 2024-04-26T01:09:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.007220565, 2024-04-26T01:09:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.332826+0000", 2024-04-26T01:09:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.332826+0000", 2024-04-26T01:09:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.332829+0000", 2024-04-26T01:09:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3130000000000001e-06 2024-04-26T01:09:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.332883+0000", 2024-04-26T01:09:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.3820000000000003e-05 2024-04-26T01:09:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.332991+0000", 2024-04-26T01:09:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107972 2024-04-26T01:09:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.333015+0000", 2024-04-26T01:09:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4287e-05 2024-04-26T01:09:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.333152+0000", 2024-04-26T01:09:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013734899999999999 2024-04-26T01:09:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.333179+0000", 2024-04-26T01:09:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7202000000000001e-05 2024-04-26T01:09:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.339853+0000", 2024-04-26T01:09:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066739340000000003 2024-04-26T01:09:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.340047+0000", 2024-04-26T01:09:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019368799999999999 2024-04-26T01:09:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.0s5 88/30 ECSubWrite(tid=109, reqid=client.4521.0:948, at_version=88'83, trim_to=0'0, roll_forward_to=81'80))", 2024-04-26T01:09:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:54.070009+0000", 2024-04-26T01:09:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 7.7742796150000002, 2024-04-26T01:09:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0083899609999999996, 2024-04-26T01:09:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.070009+0000", 2024-04-26T01:09:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.070009+0000", 2024-04-26T01:09:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.070011+0000", 2024-04-26T01:09:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.66e-06 2024-04-26T01:09:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.070055+0000", 2024-04-26T01:09:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3859000000000001e-05 2024-04-26T01:09:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.070221+0000", 2024-04-26T01:09:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016586899999999999 2024-04-26T01:09:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.070242+0000", 2024-04-26T01:09:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.1860000000000001e-05 2024-04-26T01:09:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.074413+0000", 2024-04-26T01:09:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0041703269999999997 2024-04-26T01:09:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.074439+0000", 2024-04-26T01:09:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.635e-05 2024-04-26T01:09:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.078358+0000", 2024-04-26T01:09:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0039192690000000004 2024-04-26T01:09:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.078399+0000", 2024-04-26T01:09:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.0766999999999998e-05 2024-04-26T01:09:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.0s5 88/30 ECSubWrite(tid=110, reqid=client.4521.0:949, at_version=88'84, trim_to=0'0, roll_forward_to=81'80))", 2024-04-26T01:09:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:54.070065+0000", 2024-04-26T01:09:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 7.7742236719999998, 2024-04-26T01:09:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0083670519999999998, 2024-04-26T01:09:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.070065+0000", 2024-04-26T01:09:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.070065+0000", 2024-04-26T01:09:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.070066+0000", 2024-04-26T01:09:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.8000000000000005e-07 2024-04-26T01:09:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.070134+0000", 2024-04-26T01:09:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.8446000000000002e-05 2024-04-26T01:09:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.070311+0000", 2024-04-26T01:09:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017701500000000001 2024-04-26T01:09:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.070332+0000", 2024-04-26T01:09:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.084e-05 2024-04-26T01:09:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.075952+0000", 2024-04-26T01:09:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0056202159999999999 2024-04-26T01:09:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.075967+0000", 2024-04-26T01:09:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4732999999999999e-05 2024-04-26T01:09:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.078403+0000", 2024-04-26T01:09:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0024362699999999999 2024-04-26T01:09:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.078432+0000", 2024-04-26T01:09:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8751999999999998e-05 2024-04-26T01:09:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:01.932 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:09:01.933 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:09:02.057 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:09:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:09:02.058 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:09:02.058 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:09:02.071 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:09:02.107 INFO:tasks.ceph.osd.3.smithi164.stderr:2024-04-26T01:09:02.106+0000 7fd6fea74640 -1 osd.3 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:09:02.117 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-26T01:09:02.191 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:09:02.194 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:09:02.194 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:09:02.194 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:09:02.195 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:09:02.195 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:09:02.210 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:09:02.323 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:09:02.323 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:09:02.323 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:09:02.324 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:09:02.336 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:09:02.336 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:09:02.336 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:09:02.336 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:09:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 32/30 ECSubWrite(tid=1, reqid=client.4521.0:25, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:09:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:09:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 63.022992045999999, 2024-04-26T01:09:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062762640000000001, 2024-04-26T01:09:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:09:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:09:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313833+0000", 2024-04-26T01:09:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2700000000000001e-07 2024-04-26T01:09:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.313923+0000", 2024-04-26T01:09:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.0530000000000002e-05 2024-04-26T01:09:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314064+0000", 2024-04-26T01:09:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000140841 2024-04-26T01:09:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314070+0000", 2024-04-26T01:09:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.976e-06 2024-04-26T01:09:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314140+0000", 2024-04-26T01:09:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9664000000000003e-05 2024-04-26T01:09:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.314151+0000", 2024-04-26T01:09:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1251e-05 2024-04-26T01:09:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320065+0000", 2024-04-26T01:09:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0059141990000000002 2024-04-26T01:09:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.320109+0000", 2024-04-26T01:09:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3575999999999997e-05 2024-04-26T01:09:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.ds3 33/30 ECSubWrite(tid=5, reqid=client.4521.0:35, at_version=33'4, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:09:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.325729+0000", 2024-04-26T01:09:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 63.011095386000001, 2024-04-26T01:09:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079681170000000003, 2024-04-26T01:09:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325729+0000", 2024-04-26T01:09:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325729+0000", 2024-04-26T01:09:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325730+0000", 2024-04-26T01:09:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.78e-07 2024-04-26T01:09:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325748+0000", 2024-04-26T01:09:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8127000000000001e-05 2024-04-26T01:09:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325789+0000", 2024-04-26T01:09:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.1100000000000003e-05 2024-04-26T01:09:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325800+0000", 2024-04-26T01:09:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0256e-05 2024-04-26T01:09:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325859+0000", 2024-04-26T01:09:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9849999999999998e-05 2024-04-26T01:09:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.325869+0000", 2024-04-26T01:09:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6989999999999998e-06 2024-04-26T01:09:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.333645+0000", 2024-04-26T01:09:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0077756800000000001 2024-04-26T01:09:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.333697+0000", 2024-04-26T01:09:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2426999999999998e-05 2024-04-26T01:09:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 33/30 ECSubWrite(tid=4, reqid=client.4521.0:26, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:09:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.327582+0000", 2024-04-26T01:09:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 63.009242745999998, 2024-04-26T01:09:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0074218790000000001, 2024-04-26T01:09:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327582+0000", 2024-04-26T01:09:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327582+0000", 2024-04-26T01:09:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327583+0000", 2024-04-26T01:09:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0669999999999999e-06 2024-04-26T01:09:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327596+0000", 2024-04-26T01:09:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3135e-05 2024-04-26T01:09:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327649+0000", 2024-04-26T01:09:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.2704999999999998e-05 2024-04-26T01:09:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327659+0000", 2024-04-26T01:09:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.8470000000000008e-06 2024-04-26T01:09:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327733+0000", 2024-04-26T01:09:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.4114000000000002e-05 2024-04-26T01:09:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.327746+0000", 2024-04-26T01:09:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3176999999999999e-05 2024-04-26T01:09:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.334969+0000", 2024-04-26T01:09:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0072232160000000002 2024-04-26T01:09:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.335004+0000", 2024-04-26T01:09:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4618e-05 2024-04-26T01:09:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 50/44 ECSubWrite(tid=213, reqid=osd.4.0:213, at_version=50'36, trim_to=0'0, roll_forward_to=47'34))", 2024-04-26T01:09:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.047619+0000", 2024-04-26T01:09:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 46.289205565000003, 2024-04-26T01:09:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012971391000000001, 2024-04-26T01:09:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047619+0000", 2024-04-26T01:09:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047619+0000", 2024-04-26T01:09:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047620+0000", 2024-04-26T01:09:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.9999999999999997e-07 2024-04-26T01:09:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047663+0000", 2024-04-26T01:09:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3254000000000003e-05 2024-04-26T01:09:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047762+0000", 2024-04-26T01:09:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.9551999999999998e-05 2024-04-26T01:09:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.047793+0000", 2024-04-26T01:09:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.0729999999999999e-05 2024-04-26T01:09:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.048521+0000", 2024-04-26T01:09:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000727971 2024-04-26T01:09:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.048552+0000", 2024-04-26T01:09:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1375999999999999e-05 2024-04-26T01:09:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060503+0000", 2024-04-26T01:09:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011950771000000001 2024-04-26T01:09:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.060590+0000", 2024-04-26T01:09:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.7137000000000003e-05 2024-04-26T01:09:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 50/44 ECSubWrite(tid=217, reqid=osd.4.0:213, at_version=0'0, trim_to=0'0, roll_forward_to=50'36))", 2024-04-26T01:09:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.063223+0000", 2024-04-26T01:09:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 46.273601089000003, 2024-04-26T01:09:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0071347260000000001, 2024-04-26T01:09:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063223+0000", 2024-04-26T01:09:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063223+0000", 2024-04-26T01:09:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063225+0000", 2024-04-26T01:09:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.753e-06 2024-04-26T01:09:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063249+0000", 2024-04-26T01:09:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3896999999999998e-05 2024-04-26T01:09:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063327+0000", 2024-04-26T01:09:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.7693000000000003e-05 2024-04-26T01:09:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063349+0000", 2024-04-26T01:09:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2446000000000002e-05 2024-04-26T01:09:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063511+0000", 2024-04-26T01:09:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016191799999999999 2024-04-26T01:09:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.063538+0000", 2024-04-26T01:09:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7237e-05 2024-04-26T01:09:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.070265+0000", 2024-04-26T01:09:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.006726526 2024-04-26T01:09:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.070358+0000", 2024-04-26T01:09:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.3256000000000003e-05 2024-04-26T01:09:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.1s4 50/44 ECSubWrite(tid=219, reqid=osd.4.0:219, at_version=50'58, trim_to=0'0, roll_forward_to=47'56))", 2024-04-26T01:09:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.066105+0000", 2024-04-26T01:09:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 46.270720064999999, 2024-04-26T01:09:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079801060000000007, 2024-04-26T01:09:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066105+0000", 2024-04-26T01:09:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066105+0000", 2024-04-26T01:09:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066106+0000", 2024-04-26T01:09:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.846e-06 2024-04-26T01:09:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066156+0000", 2024-04-26T01:09:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9225999999999999e-05 2024-04-26T01:09:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066238+0000", 2024-04-26T01:09:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2126999999999995e-05 2024-04-26T01:09:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066260+0000", 2024-04-26T01:09:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2408999999999998e-05 2024-04-26T01:09:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066417+0000", 2024-04-26T01:09:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000157312 2024-04-26T01:09:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.066443+0000", 2024-04-26T01:09:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5797000000000001e-05 2024-04-26T01:09:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073990+0000", 2024-04-26T01:09:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0075467670000000002 2024-04-26T01:09:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.074085+0000", 2024-04-26T01:09:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.4622000000000003e-05 2024-04-26T01:09:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 50/44 ECSubWrite(tid=220, reqid=client.4521.0:491, at_version=50'38, trim_to=0'0, roll_forward_to=50'36))", 2024-04-26T01:09:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.072911+0000", 2024-04-26T01:09:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 46.263913543000001, 2024-04-26T01:09:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0076486990000000001, 2024-04-26T01:09:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.072911+0000", 2024-04-26T01:09:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.072911+0000", 2024-04-26T01:09:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.072914+0000", 2024-04-26T01:09:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5900000000000002e-06 2024-04-26T01:09:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073741+0000", 2024-04-26T01:09:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00082785699999999999 2024-04-26T01:09:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073888+0000", 2024-04-26T01:09:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014690499999999999 2024-04-26T01:09:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.073915+0000", 2024-04-26T01:09:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.614e-05 2024-04-26T01:09:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.074074+0000", 2024-04-26T01:09:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015989500000000001 2024-04-26T01:09:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.074101+0000", 2024-04-26T01:09:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6693999999999999e-05 2024-04-26T01:09:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080467+0000", 2024-04-26T01:09:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0063662689999999999 2024-04-26T01:09:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080560+0000", 2024-04-26T01:09:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2349000000000006e-05 2024-04-26T01:09:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 50/44 ECSubWrite(tid=23, reqid=osd.9.0:22, at_version=0'0, trim_to=0'0, roll_forward_to=50'37))", 2024-04-26T01:09:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:16.079931+0000", 2024-04-26T01:09:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 46.256894029000001, 2024-04-26T01:09:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066276529999999998, 2024-04-26T01:09:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079931+0000", 2024-04-26T01:09:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079931+0000", 2024-04-26T01:09:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079933+0000", 2024-04-26T01:09:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.6759999999999999e-06 2024-04-26T01:09:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.079965+0000", 2024-04-26T01:09:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.1927000000000001e-05 2024-04-26T01:09:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080073+0000", 2024-04-26T01:09:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107796 2024-04-26T01:09:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080098+0000", 2024-04-26T01:09:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4822999999999999e-05 2024-04-26T01:09:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080244+0000", 2024-04-26T01:09:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014625799999999999 2024-04-26T01:09:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.080268+0000", 2024-04-26T01:09:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4128e-05 2024-04-26T01:09:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.086463+0000", 2024-04-26T01:09:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061951300000000001 2024-04-26T01:09:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:16.086558+0000", 2024-04-26T01:09:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.4914999999999995e-05 2024-04-26T01:09:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 53/44 ECSubWrite(tid=249, reqid=client.4521.0:539, at_version=53'46, trim_to=0'0, roll_forward_to=51'44))", 2024-04-26T01:09:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:19.133290+0000", 2024-04-26T01:09:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 43.203534511000001, 2024-04-26T01:09:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079087370000000008, 2024-04-26T01:09:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.133290+0000", 2024-04-26T01:09:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.133290+0000", 2024-04-26T01:09:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.133291+0000", 2024-04-26T01:09:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.4700000000000001e-07 2024-04-26T01:09:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.134718+0000", 2024-04-26T01:09:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001426921 2024-04-26T01:09:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.134818+0000", 2024-04-26T01:09:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000100203 2024-04-26T01:09:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.134841+0000", 2024-04-26T01:09:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2730000000000001e-05 2024-04-26T01:09:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.135022+0000", 2024-04-26T01:09:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00018121500000000001 2024-04-26T01:09:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.135051+0000", 2024-04-26T01:09:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8920000000000001e-05 2024-04-26T01:09:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.141103+0000", 2024-04-26T01:09:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0060520549999999998 2024-04-26T01:09:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:19.141199+0000", 2024-04-26T01:09:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.6045999999999995e-05 2024-04-26T01:09:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 59/44 ECSubWrite(tid=323, reqid=osd.4.0:323, at_version=59'77, trim_to=0'0, roll_forward_to=58'73))", 2024-04-26T01:09:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.160765+0000", 2024-04-26T01:09:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 37.176059492, 2024-04-26T01:09:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066099469999999997, 2024-04-26T01:09:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160765+0000", 2024-04-26T01:09:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160765+0000", 2024-04-26T01:09:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160766+0000", 2024-04-26T01:09:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.1999999999999999e-07 2024-04-26T01:09:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.160804+0000", 2024-04-26T01:09:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8068999999999998e-05 2024-04-26T01:09:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161001+0000", 2024-04-26T01:09:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000197159 2024-04-26T01:09:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.161025+0000", 2024-04-26T01:09:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3768999999999999e-05 2024-04-26T01:09:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.162206+0000", 2024-04-26T01:09:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001181398 2024-04-26T01:09:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.162230+0000", 2024-04-26T01:09:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.419e-05 2024-04-26T01:09:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167200+0000", 2024-04-26T01:09:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0049697099999999996 2024-04-26T01:09:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.167375+0000", 2024-04-26T01:09:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017493200000000001 2024-04-26T01:09:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 59/44 ECSubWrite(tid=325, reqid=osd.4.0:323, at_version=0'0, trim_to=0'0, roll_forward_to=59'77))", 2024-04-26T01:09:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:25.175746+0000", 2024-04-26T01:09:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 37.161078539000002, 2024-04-26T01:09:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0064646620000000004, 2024-04-26T01:09:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175746+0000", 2024-04-26T01:09:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175746+0000", 2024-04-26T01:09:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175748+0000", 2024-04-26T01:09:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0140000000000001e-06 2024-04-26T01:09:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175783+0000", 2024-04-26T01:09:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4946000000000001e-05 2024-04-26T01:09:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175872+0000", 2024-04-26T01:09:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8722999999999997e-05 2024-04-26T01:09:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.175895+0000", 2024-04-26T01:09:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3649999999999999e-05 2024-04-26T01:09:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.176058+0000", 2024-04-26T01:09:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00016219499999999999 2024-04-26T01:09:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.176085+0000", 2024-04-26T01:09:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7566e-05 2024-04-26T01:09:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.182118+0000", 2024-04-26T01:09:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0060333349999999999 2024-04-26T01:09:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:25.182211+0000", 2024-04-26T01:09:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2232999999999996e-05 2024-04-26T01:09:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/44 ECSubWrite(tid=353, reqid=osd.4.0:353, at_version=60'83, trim_to=0'0, roll_forward_to=59'79))", 2024-04-26T01:09:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.186337+0000", 2024-04-26T01:09:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 36.150487282, 2024-04-26T01:09:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0065831279999999997, 2024-04-26T01:09:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186337+0000", 2024-04-26T01:09:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186337+0000", 2024-04-26T01:09:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186339+0000", 2024-04-26T01:09:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.08e-06 2024-04-26T01:09:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186388+0000", 2024-04-26T01:09:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9116000000000002e-05 2024-04-26T01:09:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186438+0000", 2024-04-26T01:09:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9586000000000003e-05 2024-04-26T01:09:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186463+0000", 2024-04-26T01:09:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4859999999999999e-05 2024-04-26T01:09:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186608+0000", 2024-04-26T01:09:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000145162 2024-04-26T01:09:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.186632+0000", 2024-04-26T01:09:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4179999999999999e-05 2024-04-26T01:09:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.192817+0000", 2024-04-26T01:09:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061849080000000002 2024-04-26T01:09:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.192920+0000", 2024-04-26T01:09:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000103236 2024-04-26T01:09:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 60/44 ECSubWrite(tid=52, reqid=osd.9.0:52, at_version=60'62, trim_to=0'0, roll_forward_to=60'60))", 2024-04-26T01:09:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.205252+0000", 2024-04-26T01:09:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 36.131572679000001, 2024-04-26T01:09:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.006970324, 2024-04-26T01:09:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205252+0000", 2024-04-26T01:09:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205252+0000", 2024-04-26T01:09:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205254+0000", 2024-04-26T01:09:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8169999999999999e-06 2024-04-26T01:09:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205305+0000", 2024-04-26T01:09:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.1619000000000002e-05 2024-04-26T01:09:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205912+0000", 2024-04-26T01:09:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00060660599999999999 2024-04-26T01:09:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.205940+0000", 2024-04-26T01:09:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7596000000000001e-05 2024-04-26T01:09:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.206168+0000", 2024-04-26T01:09:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00022865200000000001 2024-04-26T01:09:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.206205+0000", 2024-04-26T01:09:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6439000000000001e-05 2024-04-26T01:09:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.212015+0000", 2024-04-26T01:09:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0058099400000000004 2024-04-26T01:09:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.212222+0000", 2024-04-26T01:09:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000207655 2024-04-26T01:09:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 71/44 ECSubWrite(tid=73, reqid=osd.9.0:73, at_version=71'81, trim_to=0'0, roll_forward_to=70'78))", 2024-04-26T01:09:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.110054+0000", 2024-04-26T01:09:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 25.226770852000001, 2024-04-26T01:09:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0070291959999999997, 2024-04-26T01:09:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110054+0000", 2024-04-26T01:09:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110054+0000", 2024-04-26T01:09:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110055+0000", 2024-04-26T01:09:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.48e-06 2024-04-26T01:09:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110105+0000", 2024-04-26T01:09:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9332999999999997e-05 2024-04-26T01:09:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110193+0000", 2024-04-26T01:09:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.8938999999999997e-05 2024-04-26T01:09:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110214+0000", 2024-04-26T01:09:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0939999999999999e-05 2024-04-26T01:09:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110505+0000", 2024-04-26T01:09:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00029090499999999999 2024-04-26T01:09:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.110521+0000", 2024-04-26T01:09:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5855999999999999e-05 2024-04-26T01:09:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.117043+0000", 2024-04-26T01:09:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0065219570000000001 2024-04-26T01:09:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.117083+0000", 2024-04-26T01:09:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.9786000000000002e-05 2024-04-26T01:09:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 71/44 ECSubWrite(tid=75, reqid=client.4521.0:731, at_version=71'83, trim_to=0'0, roll_forward_to=71'79))", 2024-04-26T01:09:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.124820+0000", 2024-04-26T01:09:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 25.212004427, 2024-04-26T01:09:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0069566530000000001, 2024-04-26T01:09:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.124820+0000", 2024-04-26T01:09:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.124820+0000", 2024-04-26T01:09:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.124822+0000", 2024-04-26T01:09:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8649999999999999e-06 2024-04-26T01:09:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.125176+0000", 2024-04-26T01:09:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00035433799999999999 2024-04-26T01:09:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.125244+0000", 2024-04-26T01:09:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.7162999999999994e-05 2024-04-26T01:09:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.125258+0000", 2024-04-26T01:09:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4418e-05 2024-04-26T01:09:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.125406+0000", 2024-04-26T01:09:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00014758499999999999 2024-04-26T01:09:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.125435+0000", 2024-04-26T01:09:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.932e-05 2024-04-26T01:09:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.131716+0000", 2024-04-26T01:09:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062809900000000002 2024-04-26T01:09:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.131777+0000", 2024-04-26T01:09:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.0974000000000003e-05 2024-04-26T01:09:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 71/44 ECSubWrite(tid=76, reqid=client.4521.0:732, at_version=71'84, trim_to=0'0, roll_forward_to=71'81))", 2024-04-26T01:09:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.127563+0000", 2024-04-26T01:09:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 25.209261145999999, 2024-04-26T01:09:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0066958829999999997, 2024-04-26T01:09:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127563+0000", 2024-04-26T01:09:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127563+0000", 2024-04-26T01:09:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127565+0000", 2024-04-26T01:09:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.17e-06 2024-04-26T01:09:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127666+0000", 2024-04-26T01:09:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000100997 2024-04-26T01:09:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127710+0000", 2024-04-26T01:09:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.4496000000000002e-05 2024-04-26T01:09:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127722+0000", 2024-04-26T01:09:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2252000000000001e-05 2024-04-26T01:09:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127954+0000", 2024-04-26T01:09:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00023175299999999999 2024-04-26T01:09:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.127977+0000", 2024-04-26T01:09:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3093000000000001e-05 2024-04-26T01:09:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.134210+0000", 2024-04-26T01:09:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062328790000000002 2024-04-26T01:09:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.134259+0000", 2024-04-26T01:09:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9243000000000003e-05 2024-04-26T01:09:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 77/44 ECSubWrite(tid=89, reqid=client.4521.0:870, at_version=77'92, trim_to=0'0, roll_forward_to=74'89))", 2024-04-26T01:09:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:43.346881+0000", 2024-04-26T01:09:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 18.989943313000001, 2024-04-26T01:09:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.071918344999999995, 2024-04-26T01:09:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.346881+0000", 2024-04-26T01:09:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.346881+0000", 2024-04-26T01:09:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.346883+0000", 2024-04-26T01:09:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7099999999999999e-06 2024-04-26T01:09:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.346926+0000", 2024-04-26T01:09:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.2656e-05 2024-04-26T01:09:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.417353+0000", 2024-04-26T01:09:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.070427139999999999 2024-04-26T01:09:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.417380+0000", 2024-04-26T01:09:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.7586999999999999e-05 2024-04-26T01:09:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.417538+0000", 2024-04-26T01:09:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000157429 2024-04-26T01:09:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.417556+0000", 2024-04-26T01:09:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8436000000000001e-05 2024-04-26T01:09:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.418726+0000", 2024-04-26T01:09:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0011697400000000001 2024-04-26T01:09:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:43.418800+0000", 2024-04-26T01:09:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.3646999999999995e-05 2024-04-26T01:09:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.1s4 81/44 ECSubWrite(tid=477, reqid=client.4521.0:895, at_version=81'99, trim_to=0'0, roll_forward_to=81'97))", 2024-04-26T01:09:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:47.483962+0000", 2024-04-26T01:09:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 14.852862102, 2024-04-26T01:09:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0068314680000000003, 2024-04-26T01:09:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.483962+0000", 2024-04-26T01:09:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.483962+0000", 2024-04-26T01:09:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.483964+0000", 2024-04-26T01:09:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6169999999999999e-06 2024-04-26T01:09:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.484339+0000", 2024-04-26T01:09:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000374545 2024-04-26T01:09:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.484397+0000", 2024-04-26T01:09:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.8267999999999998e-05 2024-04-26T01:09:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.484410+0000", 2024-04-26T01:09:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3061000000000001e-05 2024-04-26T01:09:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.484487+0000", 2024-04-26T01:09:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.7156999999999993e-05 2024-04-26T01:09:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.484498+0000", 2024-04-26T01:09:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0983e-05 2024-04-26T01:09:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.490751+0000", 2024-04-26T01:09:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0062530249999999997 2024-04-26T01:09:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.490794+0000", 2024-04-26T01:09:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.2812000000000003e-05 2024-04-26T01:09:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.1s4 83/44 ECSubWrite(tid=496, reqid=osd.4.0:496, at_version=83'105, trim_to=0'0, roll_forward_to=81'101))", 2024-04-26T01:09:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:49.328045+0000", 2024-04-26T01:09:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 13.008779883000001, 2024-04-26T01:09:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0074025810000000001, 2024-04-26T01:09:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.328045+0000", 2024-04-26T01:09:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.328045+0000", 2024-04-26T01:09:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.328047+0000", 2024-04-26T01:09:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.2069999999999998e-06 2024-04-26T01:09:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.328097+0000", 2024-04-26T01:09:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0139999999999998e-05 2024-04-26T01:09:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.328176+0000", 2024-04-26T01:09:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.8602000000000004e-05 2024-04-26T01:09:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.328198+0000", 2024-04-26T01:09:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.281e-05 2024-04-26T01:09:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.329482+0000", 2024-04-26T01:09:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0012835069999999999 2024-04-26T01:09:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.329510+0000", 2024-04-26T01:09:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8246e-05 2024-04-26T01:09:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.335353+0000", 2024-04-26T01:09:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0058423670000000002 2024-04-26T01:09:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:49.335447+0000", 2024-04-26T01:09:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.4702000000000002e-05 2024-04-26T01:09:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.cs3 88/44 ECSubWrite(tid=111, reqid=client.4521.0:950, at_version=88'106, trim_to=0'0, roll_forward_to=86'104))", 2024-04-26T01:09:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:54.078820+0000", 2024-04-26T01:09:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 8.2580048050000006, 2024-04-26T01:09:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0069099620000000004, 2024-04-26T01:09:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.078820+0000", 2024-04-26T01:09:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.078820+0000", 2024-04-26T01:09:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.078821+0000", 2024-04-26T01:09:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.54e-07 2024-04-26T01:09:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.079102+0000", 2024-04-26T01:09:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00028148000000000002 2024-04-26T01:09:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.079200+0000", 2024-04-26T01:09:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7399999999999996e-05 2024-04-26T01:09:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.079225+0000", 2024-04-26T01:09:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5669000000000001e-05 2024-04-26T01:09:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.079410+0000", 2024-04-26T01:09:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00018513899999999999 2024-04-26T01:09:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.079439+0000", 2024-04-26T01:09:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8702999999999999e-05 2024-04-26T01:09:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.085637+0000", 2024-04-26T01:09:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0061982740000000001 2024-04-26T01:09:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.085730+0000", 2024-04-26T01:09:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2442999999999996e-05 2024-04-26T01:09:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.428 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:09:02.429 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed 2024-04-26T01:09:02.430 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:09:02.556 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:09:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:09:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "num_ops": 0 2024-04-26T01:09:02.556 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:09:02.570 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:09:02.692 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:09:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [], 2024-04-26T01:09:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "complaint_time": 30, 2024-04-26T01:09:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "num_blocked_ops": 0 2024-04-26T01:09:02.692 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:09:02.706 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:09:02.833 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-26T01:09:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "size": 20, 2024-04-26T01:09:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 600, 2024-04-26T01:09:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "ops": [ 2024-04-26T01:09:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:105 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:09:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 63.405928451999998, 2024-04-26T01:09:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017139178000000001, 2024-04-26T01:09:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:09:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 105 2024-04-26T01:09:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:09:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425520+0000", 2024-04-26T01:09:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425522+0000", 2024-04-26T01:09:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5430000000000001e-06 2024-04-26T01:09:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425529+0000", 2024-04-26T01:09:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2570000000000001e-06 2024-04-26T01:09:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425548+0000", 2024-04-26T01:09:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.8674000000000001e-05 2024-04-26T01:09:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.425560+0000", 2024-04-26T01:09:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2683e-05 2024-04-26T01:09:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428152+0000", 2024-04-26T01:09:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0025912800000000001 2024-04-26T01:09:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.428234+0000", 2024-04-26T01:09:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2105000000000001e-05 2024-04-26T01:09:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438095+0000", 2024-04-26T01:09:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0098614180000000003 2024-04-26T01:09:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438205+0000", 2024-04-26T01:09:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000109849 2024-04-26T01:09:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442659+0000", 2024-04-26T01:09:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0044543689999999997 2024-04-26T01:09:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:117 3.0s0 3:00e538ad:::smithi16428812-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:09:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 63.392825455999997, 2024-04-26T01:09:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017480893000000001, 2024-04-26T01:09:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:09:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 117 2024-04-26T01:09:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:09:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438623+0000", 2024-04-26T01:09:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438624+0000", 2024-04-26T01:09:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.6000000000000003e-07 2024-04-26T01:09:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438628+0000", 2024-04-26T01:09:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8029999999999998e-06 2024-04-26T01:09:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438635+0000", 2024-04-26T01:09:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2799999999999998e-06 2024-04-26T01:09:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.438641+0000", 2024-04-26T01:09:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5459999999999997e-06 2024-04-26T01:09:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442203+0000", 2024-04-26T01:09:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.003561594 2024-04-26T01:09:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.442287+0000", 2024-04-26T01:09:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.4443000000000004e-05 2024-04-26T01:09:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451430+0000", 2024-04-26T01:09:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0091427130000000002 2024-04-26T01:09:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.451500+0000", 2024-04-26T01:09:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9342000000000001e-05 2024-04-26T01:09:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.456104+0000", 2024-04-26T01:09:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0046045119999999998 2024-04-26T01:09:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:153 3.0s0 3:0230bbd3:::smithi16428812-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:09:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 63.342102294, 2024-04-26T01:09:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017766607, 2024-04-26T01:09:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:09:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 153 2024-04-26T01:09:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:09:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489346+0000", 2024-04-26T01:09:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489347+0000", 2024-04-26T01:09:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.9999999999999997e-07 2024-04-26T01:09:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489351+0000", 2024-04-26T01:09:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7309999999999999e-06 2024-04-26T01:09:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489359+0000", 2024-04-26T01:09:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.6619999999999994e-06 2024-04-26T01:09:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.489366+0000", 2024-04-26T01:09:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.798e-06 2024-04-26T01:09:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492454+0000", 2024-04-26T01:09:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0030882399999999999 2024-04-26T01:09:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.492562+0000", 2024-04-26T01:09:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000107948 2024-04-26T01:09:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.504997+0000", 2024-04-26T01:09:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.012434176 2024-04-26T01:09:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.505136+0000", 2024-04-26T01:09:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013994300000000001 2024-04-26T01:09:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:07:59.507113+0000", 2024-04-26T01:09:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0019764090000000002 2024-04-26T01:09:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:287 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-26T01:09:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:04.533969+0000", 2024-04-26T01:09:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 58.297479223000003, 2024-04-26T01:09:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.018210147999999999, 2024-04-26T01:09:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:09:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 287 2024-04-26T01:09:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533969+0000", 2024-04-26T01:09:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533969+0000", 2024-04-26T01:09:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533971+0000", 2024-04-26T01:09:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3629999999999999e-06 2024-04-26T01:09:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.533977+0000", 2024-04-26T01:09:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.5799999999999997e-06 2024-04-26T01:09:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.534001+0000", 2024-04-26T01:09:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.4006999999999999e-05 2024-04-26T01:09:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.534016+0000", 2024-04-26T01:09:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5160000000000001e-05 2024-04-26T01:09:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.545071+0000", 2024-04-26T01:09:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.011054938 2024-04-26T01:09:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.545349+0000", 2024-04-26T01:09:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00027734100000000001 2024-04-26T01:09:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.549341+0000", 2024-04-26T01:09:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0039925150000000003 2024-04-26T01:09:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.549639+0000", 2024-04-26T01:09:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00029756800000000001 2024-04-26T01:09:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:04.552180+0000", 2024-04-26T01:09:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.002540676 2024-04-26T01:09:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:542 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~606208] snapc e=[e,d,c,b,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-04-26T01:09:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:20.149139+0000", 2024-04-26T01:09:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 42.682309648999997, 2024-04-26T01:09:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016978235000000001, 2024-04-26T01:09:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 542 2024-04-26T01:09:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.149139+0000", 2024-04-26T01:09:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.149139+0000", 2024-04-26T01:09:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.149142+0000", 2024-04-26T01:09:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.9639999999999999e-06 2024-04-26T01:09:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150216+0000", 2024-04-26T01:09:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001074478 2024-04-26T01:09:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150251+0000", 2024-04-26T01:09:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4563000000000001e-05 2024-04-26T01:09:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150269+0000", 2024-04-26T01:09:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.806e-05 2024-04-26T01:09:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150398+0000", 2024-04-26T01:09:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00012864500000000001 2024-04-26T01:09:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "op must wait for map", 2024-04-26T01:09:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.150410+0000", 2024-04-26T01:09:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2427e-05 2024-04-26T01:09:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.154650+0000", 2024-04-26T01:09:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0042403129999999999 2024-04-26T01:09:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.154806+0000", 2024-04-26T01:09:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015539199999999999 2024-04-26T01:09:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.157843+0000", 2024-04-26T01:09:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0030374519999999999 2024-04-26T01:09:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.165692+0000", 2024-04-26T01:09:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0078487009999999996 2024-04-26T01:09:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:09:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.165984+0000", 2024-04-26T01:09:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00029256099999999999 2024-04-26T01:09:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:20.166117+0000", 2024-04-26T01:09:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013267899999999999 2024-04-26T01:09:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.4s5 55/30 ECSubWrite(tid=39, reqid=osd.8.0:39, at_version=55'26, trim_to=0'0, roll_forward_to=52'24))", 2024-04-26T01:09:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:21.091694+0000", 2024-04-26T01:09:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 41.739754366, 2024-04-26T01:09:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.49007189200000001, 2024-04-26T01:09:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091694+0000", 2024-04-26T01:09:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091694+0000", 2024-04-26T01:09:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091695+0000", 2024-04-26T01:09:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.0699999999999996e-07 2024-04-26T01:09:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.091753+0000", 2024-04-26T01:09:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.7516999999999998e-05 2024-04-26T01:09:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580278+0000", 2024-04-26T01:09:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.48852570099999998 2024-04-26T01:09:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580292+0000", 2024-04-26T01:09:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3671e-05 2024-04-26T01:09:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580396+0000", 2024-04-26T01:09:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00010389400000000001 2024-04-26T01:09:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.580413+0000", 2024-04-26T01:09:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7121000000000001e-05 2024-04-26T01:09:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.581731+0000", 2024-04-26T01:09:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001318294 2024-04-26T01:09:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:21.581766+0000", 2024-04-26T01:09:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.4786999999999997e-05 2024-04-26T01:09:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 58/30 ECSubWriteReply(tid=65, last_complete=58'49, committed=1, applied=1))", 2024-04-26T01:09:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:24.344800+0000", 2024-04-26T01:09:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 38.486648658999997, 2024-04-26T01:09:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.33836068000000002, 2024-04-26T01:09:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344800+0000", 2024-04-26T01:09:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344800+0000", 2024-04-26T01:09:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344802+0000", 2024-04-26T01:09:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.593e-06 2024-04-26T01:09:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.344812+0000", 2024-04-26T01:09:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0943e-05 2024-04-26T01:09:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.682937+0000", 2024-04-26T01:09:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.33812443599999997 2024-04-26T01:09:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.682952+0000", 2024-04-26T01:09:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5116000000000001e-05 2024-04-26T01:09:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.683088+0000", 2024-04-26T01:09:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000135969 2024-04-26T01:09:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:24.683161+0000", 2024-04-26T01:09:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.2622999999999999e-05 2024-04-26T01:09:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "MOSDECSubOpWrite(3.4s5 60/30 ECSubWrite(tid=58, reqid=osd.8.0:58, at_version=60'44, trim_to=0'0, roll_forward_to=59'40))", 2024-04-26T01:09:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:26.181832+0000", 2024-04-26T01:09:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 36.649616438999999, 2024-04-26T01:09:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.020381277999999999, 2024-04-26T01:09:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "reached pg", 2024-04-26T01:09:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181832+0000", 2024-04-26T01:09:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181832+0000", 2024-04-26T01:09:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181833+0000", 2024-04-26T01:09:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.0999999999999998e-07 2024-04-26T01:09:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.181871+0000", 2024-04-26T01:09:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8352999999999997e-05 2024-04-26T01:09:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.182274+0000", 2024-04-26T01:09:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00040273700000000002 2024-04-26T01:09:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.182291+0000", 2024-04-26T01:09:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6949999999999999e-05 2024-04-26T01:09:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.187997+0000", 2024-04-26T01:09:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.005706198 2024-04-26T01:09:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.188021+0000", 2024-04-26T01:09:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3652999999999999e-05 2024-04-26T01:09:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.202131+0000", 2024-04-26T01:09:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.014110531000000001 2024-04-26T01:09:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:26.202213+0000", 2024-04-26T01:09:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.2145999999999996e-05 2024-04-26T01:09:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:770 3.0s0 3:0982979f:::smithi16428812-35:head [append 0~671744] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:09:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.264210+0000", 2024-04-26T01:09:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 25.567238121999999, 2024-04-26T01:09:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.016944765000000001, 2024-04-26T01:09:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 770 2024-04-26T01:09:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.264210+0000", 2024-04-26T01:09:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.264210+0000", 2024-04-26T01:09:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.264214+0000", 2024-04-26T01:09:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.2799999999999999e-06 2024-04-26T01:09:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.265140+0000", 2024-04-26T01:09:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00092632999999999995 2024-04-26T01:09:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.265176+0000", 2024-04-26T01:09:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6253e-05 2024-04-26T01:09:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.265192+0000", 2024-04-26T01:09:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5792999999999999e-05 2024-04-26T01:09:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.265340+0000", 2024-04-26T01:09:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000148212 2024-04-26T01:09:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.265514+0000", 2024-04-26T01:09:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00017378200000000001 2024-04-26T01:09:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.267348+0000", 2024-04-26T01:09:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0018339319999999999 2024-04-26T01:09:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.280762+0000", 2024-04-26T01:09:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.01341373 2024-04-26T01:09:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:09:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.280956+0000", 2024-04-26T01:09:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019426400000000001 2024-04-26T01:09:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.281155+0000", 2024-04-26T01:09:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019918900000000001 2024-04-26T01:09:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:774 3.0s0 3:0982979f:::smithi16428812-35:head [setxattr _header (57) in=7b] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:09:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.273352+0000", 2024-04-26T01:09:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 25.558096578000001, 2024-04-26T01:09:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.31518492300000001, 2024-04-26T01:09:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 774 2024-04-26T01:09:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273352+0000", 2024-04-26T01:09:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273352+0000", 2024-04-26T01:09:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273353+0000", 2024-04-26T01:09:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.407e-06 2024-04-26T01:09:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273361+0000", 2024-04-26T01:09:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.8960000000000003e-06 2024-04-26T01:09:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273382+0000", 2024-04-26T01:09:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.0564000000000002e-05 2024-04-26T01:09:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273397+0000", 2024-04-26T01:09:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5492999999999999e-05 2024-04-26T01:09:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.282682+0000", 2024-04-26T01:09:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0092849830000000001 2024-04-26T01:09:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.282844+0000", 2024-04-26T01:09:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000161562 2024-04-26T01:09:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.283651+0000", 2024-04-26T01:09:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00080674999999999996 2024-04-26T01:09:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.285414+0000", 2024-04-26T01:09:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0017628800000000001 2024-04-26T01:09:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:09:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.588422+0000", 2024-04-26T01:09:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.30300855599999998 2024-04-26T01:09:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.588537+0000", 2024-04-26T01:09:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000114832 2024-04-26T01:09:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:775 3.0s0 3:0982979f:::smithi16428812-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:09:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:37.273465+0000", 2024-04-26T01:09:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 25.557984058999999, 2024-04-26T01:09:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.31904869699999999, 2024-04-26T01:09:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:09:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 775 2024-04-26T01:09:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273465+0000", 2024-04-26T01:09:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273465+0000", 2024-04-26T01:09:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273466+0000", 2024-04-26T01:09:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4929999999999999e-06 2024-04-26T01:09:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273472+0000", 2024-04-26T01:09:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.2040000000000002e-06 2024-04-26T01:09:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273487+0000", 2024-04-26T01:09:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.4486e-05 2024-04-26T01:09:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.273497+0000", 2024-04-26T01:09:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0020000000000001e-05 2024-04-26T01:09:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.284233+0000", 2024-04-26T01:09:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010736104999999999 2024-04-26T01:09:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.284468+0000", 2024-04-26T01:09:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000235508 2024-04-26T01:09:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.589239+0000", 2024-04-26T01:09:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.30477114199999999 2024-04-26T01:09:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.589355+0000", 2024-04-26T01:09:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000115987 2024-04-26T01:09:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:37.592513+0000", 2024-04-26T01:09:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0031577520000000002 2024-04-26T01:09:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:797 3.0s0 3:0d858d01:::smithi16428812-41:head [append 0~720896] snapc 16=[16,15] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-26T01:09:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:39.267373+0000", 2024-04-26T01:09:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 23.564075406000001, 2024-04-26T01:09:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.019993444999999999, 2024-04-26T01:09:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 797 2024-04-26T01:09:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.267373+0000", 2024-04-26T01:09:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.267373+0000", 2024-04-26T01:09:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.267377+0000", 2024-04-26T01:09:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.7869999999999999e-06 2024-04-26T01:09:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.269934+0000", 2024-04-26T01:09:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0025572030000000001 2024-04-26T01:09:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.269981+0000", 2024-04-26T01:09:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.6409999999999998e-05 2024-04-26T01:09:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.270004+0000", 2024-04-26T01:09:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3563000000000001e-05 2024-04-26T01:09:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.270160+0000", 2024-04-26T01:09:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015629899999999999 2024-04-26T01:09:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.270398+0000", 2024-04-26T01:09:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000237915 2024-04-26T01:09:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.273343+0000", 2024-04-26T01:09:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0029442240000000001 2024-04-26T01:09:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.281421+0000", 2024-04-26T01:09:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0080788019999999995 2024-04-26T01:09:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:09:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.287233+0000", 2024-04-26T01:09:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0058111930000000001 2024-04-26T01:09:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.287367+0000", 2024-04-26T01:09:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00013404899999999999 2024-04-26T01:09:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:801 3.0s0 3:0d858d01:::smithi16428812-41:head [setxattr _header (57) in=7b] snapc 16=[16,15] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-26T01:09:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:39.282568+0000", 2024-04-26T01:09:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 23.548880989000001, 2024-04-26T01:09:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.018474469, 2024-04-26T01:09:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 801 2024-04-26T01:09:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282568+0000", 2024-04-26T01:09:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282568+0000", 2024-04-26T01:09:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282569+0000", 2024-04-26T01:09:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7719999999999999e-06 2024-04-26T01:09:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282580+0000", 2024-04-26T01:09:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.0562e-05 2024-04-26T01:09:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282598+0000", 2024-04-26T01:09:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7960000000000001e-05 2024-04-26T01:09:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282612+0000", 2024-04-26T01:09:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.3614e-05 2024-04-26T01:09:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.289835+0000", 2024-04-26T01:09:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0072236469999999997 2024-04-26T01:09:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.290022+0000", 2024-04-26T01:09:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00018657600000000001 2024-04-26T01:09:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.290813+0000", 2024-04-26T01:09:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00079124600000000001 2024-04-26T01:09:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.292480+0000", 2024-04-26T01:09:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001667237 2024-04-26T01:09:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:09:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.300923+0000", 2024-04-26T01:09:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0084426929999999994 2024-04-26T01:09:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.301042+0000", 2024-04-26T01:09:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000119162 2024-04-26T01:09:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:802 3.0s0 3:0d858d01:::smithi16428812-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-04-26T01:09:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:39.282634+0000", 2024-04-26T01:09:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 23.548815008999998, 2024-04-26T01:09:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.025775884999999998, 2024-04-26T01:09:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:09:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 802 2024-04-26T01:09:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282634+0000", 2024-04-26T01:09:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282634+0000", 2024-04-26T01:09:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282635+0000", 2024-04-26T01:09:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.2619999999999999e-06 2024-04-26T01:09:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282641+0000", 2024-04-26T01:09:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.2319999999999998e-06 2024-04-26T01:09:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282658+0000", 2024-04-26T01:09:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7274000000000001e-05 2024-04-26T01:09:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.282670+0000", 2024-04-26T01:09:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.1243000000000001e-05 2024-04-26T01:09:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.292820+0000", 2024-04-26T01:09:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.010150899999999999 2024-04-26T01:09:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.293017+0000", 2024-04-26T01:09:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019624200000000001 2024-04-26T01:09:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.301690+0000", 2024-04-26T01:09:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0086736279999999992 2024-04-26T01:09:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.301945+0000", 2024-04-26T01:09:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000254885 2024-04-26T01:09:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:39.308409+0000", 2024-04-26T01:09:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0064642190000000002 2024-04-26T01:09:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:889 3.0s0 3:0d858d01:::smithi16428812-41:18 [read 0~5275648,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:09:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:47.478482+0000", 2024-04-26T01:09:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.352966771, 2024-04-26T01:09:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.018091233000000002, 2024-04-26T01:09:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:09:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 889 2024-04-26T01:09:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478482+0000", 2024-04-26T01:09:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478482+0000", 2024-04-26T01:09:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478483+0000", 2024-04-26T01:09:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5239999999999999e-06 2024-04-26T01:09:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478493+0000", 2024-04-26T01:09:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.7389999999999994e-06 2024-04-26T01:09:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478510+0000", 2024-04-26T01:09:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.6549999999999999e-05 2024-04-26T01:09:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478519+0000", 2024-04-26T01:09:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.4450000000000002e-06 2024-04-26T01:09:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478604+0000", 2024-04-26T01:09:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 8.4685000000000007e-05 2024-04-26T01:09:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478738+0000", 2024-04-26T01:09:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000133967 2024-04-26T01:09:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.496573+0000", 2024-04-26T01:09:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017835323 2024-04-26T01:09:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:890 3.0s0 3:0d858d01:::smithi16428812-41:18 [read 0~5275648,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:09:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:47.478539+0000", 2024-04-26T01:09:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.352909344, 2024-04-26T01:09:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.021820148000000001, 2024-04-26T01:09:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:09:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 890 2024-04-26T01:09:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478539+0000", 2024-04-26T01:09:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478539+0000", 2024-04-26T01:09:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478540+0000", 2024-04-26T01:09:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3799999999999998e-07 2024-04-26T01:09:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478542+0000", 2024-04-26T01:09:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.8090000000000001e-06 2024-04-26T01:09:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478548+0000", 2024-04-26T01:09:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.04e-06 2024-04-26T01:09:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478551+0000", 2024-04-26T01:09:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.8700000000000002e-06 2024-04-26T01:09:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.479336+0000", 2024-04-26T01:09:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00078435199999999999 2024-04-26T01:09:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.479428+0000", 2024-04-26T01:09:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.2052000000000006e-05 2024-04-26T01:09:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.500359+0000", 2024-04-26T01:09:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.020931587000000001 2024-04-26T01:09:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:891 3.0s0 3:0d858d01:::smithi16428812-41:18 [read 0~5275648,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2024-04-26T01:09:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:47.478584+0000", 2024-04-26T01:09:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 15.352864370000001, 2024-04-26T01:09:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.025384917, 2024-04-26T01:09:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "started", 2024-04-26T01:09:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 891 2024-04-26T01:09:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478584+0000", 2024-04-26T01:09:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478584+0000", 2024-04-26T01:09:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478585+0000", 2024-04-26T01:09:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.3500000000000002e-07 2024-04-26T01:09:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478587+0000", 2024-04-26T01:09:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5859999999999999e-06 2024-04-26T01:09:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478594+0000", 2024-04-26T01:09:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 6.4799999999999998e-06 2024-04-26T01:09:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.478598+0000", 2024-04-26T01:09:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.07e-06 2024-04-26T01:09:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.479740+0000", 2024-04-26T01:09:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0011423010000000001 2024-04-26T01:09:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.480024+0000", 2024-04-26T01:09:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00028346399999999998 2024-04-26T01:09:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:47.503969+0000", 2024-04-26T01:09:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.023945581000000001 2024-04-26T01:09:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:947 3.0s0 3:0d858d01:::smithi16428812-41:head [append 0~0] snapc 1e=[1d,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-26T01:09:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:54.058284+0000", 2024-04-26T01:09:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 8.7731646080000001, 2024-04-26T01:09:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.017367719, 2024-04-26T01:09:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 947 2024-04-26T01:09:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058284+0000", 2024-04-26T01:09:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058284+0000", 2024-04-26T01:09:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058288+0000", 2024-04-26T01:09:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 3.6440000000000002e-06 2024-04-26T01:09:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058303+0000", 2024-04-26T01:09:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.5056e-05 2024-04-26T01:09:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058353+0000", 2024-04-26T01:09:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.9855999999999998e-05 2024-04-26T01:09:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058378+0000", 2024-04-26T01:09:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.5009999999999999e-05 2024-04-26T01:09:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058611+0000", 2024-04-26T01:09:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00023349500000000001 2024-04-26T01:09:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "op must wait for map", 2024-04-26T01:09:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058634+0000", 2024-04-26T01:09:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 2.3226000000000001e-05 2024-04-26T01:09:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.061949+0000", 2024-04-26T01:09:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0033148560000000001 2024-04-26T01:09:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.062071+0000", 2024-04-26T01:09:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000121414 2024-04-26T01:09:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.062971+0000", 2024-04-26T01:09:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00090063100000000004 2024-04-26T01:09:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.070859+0000", 2024-04-26T01:09:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0078880219999999997 2024-04-26T01:09:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:09:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.075455+0000", 2024-04-26T01:09:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0045962219999999996 2024-04-26T01:09:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.075652+0000", 2024-04-26T01:09:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00019628700000000001 2024-04-26T01:09:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:948 3.0s0 3:0d858d01:::smithi16428812-41:head [rollback 1d] snapc 1e=[1d,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-26T01:09:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:54.058432+0000", 2024-04-26T01:09:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 8.7730167219999995, 2024-04-26T01:09:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.020479824000000001, 2024-04-26T01:09:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 948 2024-04-26T01:09:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058432+0000", 2024-04-26T01:09:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058432+0000", 2024-04-26T01:09:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058433+0000", 2024-04-26T01:09:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.42e-06 2024-04-26T01:09:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058441+0000", 2024-04-26T01:09:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.5100000000000001e-06 2024-04-26T01:09:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058492+0000", 2024-04-26T01:09:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 5.0896000000000002e-05 2024-04-26T01:09:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058509+0000", 2024-04-26T01:09:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.7119999999999999e-05 2024-04-26T01:09:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058774+0000", 2024-04-26T01:09:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00026504099999999999 2024-04-26T01:09:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:09:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058781+0000", 2024-04-26T01:09:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.1459999999999999e-06 2024-04-26T01:09:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.064161+0000", 2024-04-26T01:09:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0053797519999999998 2024-04-26T01:09:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.064254+0000", 2024-04-26T01:09:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.3033000000000002e-05 2024-04-26T01:09:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.064869+0000", 2024-04-26T01:09:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00061545899999999995 2024-04-26T01:09:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.072862+0000", 2024-04-26T01:09:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0079924909999999991 2024-04-26T01:09:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:09:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.078761+0000", 2024-04-26T01:09:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0058996070000000003 2024-04-26T01:09:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.078912+0000", 2024-04-26T01:09:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00015034900000000001 2024-04-26T01:09:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "description": "osd_op(client.4521.0:949 3.0s0 3:0d858d01:::smithi16428812-41:head [append 0~0] snapc 1e=[1d,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-26T01:09:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "initiated_at": "2024-04-26T01:08:54.058533+0000", 2024-04-26T01:09:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "age": 8.7729157929999992, 2024-04-26T01:09:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.020849146999999998, 2024-04-26T01:09:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "continuous": false, 2024-04-26T01:09:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "type_data": { 2024-04-26T01:09:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "client_info": { 2024-04-26T01:09:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "client": "client.4521", 2024-04-26T01:09:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "tid": 949 2024-04-26T01:09:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "events": [ 2024-04-26T01:09:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "initiated", 2024-04-26T01:09:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058533+0000", 2024-04-26T01:09:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "header_read", 2024-04-26T01:09:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058533+0000", 2024-04-26T01:09:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0 2024-04-26T01:09:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "throttled", 2024-04-26T01:09:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058534+0000", 2024-04-26T01:09:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 9.1999999999999998e-07 2024-04-26T01:09:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "all_read", 2024-04-26T01:09:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058538+0000", 2024-04-26T01:09:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 4.6870000000000002e-06 2024-04-26T01:09:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "dispatched", 2024-04-26T01:09:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058549+0000", 2024-04-26T01:09:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.048e-05 2024-04-26T01:09:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "queued_for_pg", 2024-04-26T01:09:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058557+0000", 2024-04-26T01:09:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 7.7000000000000008e-06 2024-04-26T01:09:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058873+0000", 2024-04-26T01:09:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00031644699999999997 2024-04-26T01:09:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:09:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.058892+0000", 2024-04-26T01:09:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 1.927e-05 2024-04-26T01:09:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "reached_pg", 2024-04-26T01:09:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.068031+0000", 2024-04-26T01:09:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0091391370000000003 2024-04-26T01:09:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "started", 2024-04-26T01:09:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.068307+0000", 2024-04-26T01:09:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.00027563400000000002 2024-04-26T01:09:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_started", 2024-04-26T01:09:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.069597+0000", 2024-04-26T01:09:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.001289803 2024-04-26T01:09:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "sub_op_committed", 2024-04-26T01:09:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.075671+0000", 2024-04-26T01:09:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0060739000000000001 2024-04-26T01:09:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "commit_sent", 2024-04-26T01:09:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.079183+0000", 2024-04-26T01:09:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.0035121179999999998 2024-04-26T01:09:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-26T01:09:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-26T01:09:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "event": "done", 2024-04-26T01:09:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "time": "2024-04-26T01:08:54.079382+0000", 2024-04-26T01:09:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "duration": 0.000199051 2024-04-26T01:09:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-26T01:09:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-04-26T01:09:02.938 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-26T01:09:02.939 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:09:03.069 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:09:03.069 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:09:03.069 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:09:03.070 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:09:03.084 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:09:03.093 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-04-26T01:09:03.118 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-26T01:09:03.207 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:09:03.207 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:09:03.207 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2024-04-26T01:09:03.207 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2024-04-26T01:09:03.207 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:09:03.221 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:09:03.349 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:09:03.349 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2024-04-26T01:09:03.349 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2024-04-26T01:09:03.349 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2024-04-26T01:09:03.349 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.349 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:4 3.ds0 3:bcd9cb81:::smithi16428812-2:head [append 0~32768,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:09:03.349 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.302987+0000", 2024-04-26T01:09:03.350 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.044473834000001, 2024-04-26T01:09:03.350 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012685460000000001, 2024-04-26T01:09:03.350 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.350 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.350 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:03.350 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.350 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.350 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.350 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 4 2024-04-26T01:09:03.350 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.350 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.350 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.350 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.350 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.302987+0000", 2024-04-26T01:09:03.351 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.351 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.351 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.351 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.351 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.302987+0000", 2024-04-26T01:09:03.351 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.351 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.351 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.351 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.351 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.302989+0000", 2024-04-26T01:09:03.351 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.266e-06 2024-04-26T01:09:03.351 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.351 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.351 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.352 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303040+0000", 2024-04-26T01:09:03.352 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1029000000000002e-05 2024-04-26T01:09:03.352 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.352 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.352 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.352 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303059+0000", 2024-04-26T01:09:03.352 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8909000000000001e-05 2024-04-26T01:09:03.352 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.352 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.352 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.352 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303074+0000", 2024-04-26T01:09:03.352 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4812e-05 2024-04-26T01:09:03.352 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.352 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.353 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.353 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303197+0000", 2024-04-26T01:09:03.353 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000122748 2024-04-26T01:09:03.353 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.353 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.353 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.353 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303424+0000", 2024-04-26T01:09:03.353 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022748500000000001 2024-04-26T01:09:03.353 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.353 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.353 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.353 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.304556+0000", 2024-04-26T01:09:03.353 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001131948 2024-04-26T01:09:03.354 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.354 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.354 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.354 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.313947+0000", 2024-04-26T01:09:03.354 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0093914069999999992 2024-04-26T01:09:03.354 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.354 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.354 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:09:03.355 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315465+0000", 2024-04-26T01:09:03.355 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001517673 2024-04-26T01:09:03.355 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.355 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.355 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.355 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.315672+0000", 2024-04-26T01:09:03.355 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000207183 2024-04-26T01:09:03.355 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.355 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.355 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.355 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.355 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.356 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:5 3.ds0 3:bcd9cb81:::smithi16428812-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:09:03.356 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.303108+0000", 2024-04-26T01:09:03.356 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.044352836000002, 2024-04-26T01:09:03.356 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013204789, 2024-04-26T01:09:03.356 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.356 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.356 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:03.356 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.356 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.356 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.356 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 5 2024-04-26T01:09:03.356 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.356 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.356 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.357 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.357 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303108+0000", 2024-04-26T01:09:03.357 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.357 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.357 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.357 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.357 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303108+0000", 2024-04-26T01:09:03.357 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.357 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.357 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.357 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.357 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303109+0000", 2024-04-26T01:09:03.357 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3149999999999999e-06 2024-04-26T01:09:03.357 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.358 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.358 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.358 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303117+0000", 2024-04-26T01:09:03.358 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1899999999999995e-06 2024-04-26T01:09:03.358 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.358 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.358 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.358 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303131+0000", 2024-04-26T01:09:03.358 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3890000000000001e-05 2024-04-26T01:09:03.358 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.358 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.358 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.358 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303143+0000", 2024-04-26T01:09:03.358 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1773999999999999e-05 2024-04-26T01:09:03.359 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.359 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.359 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.359 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.305710+0000", 2024-04-26T01:09:03.359 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0025668930000000002 2024-04-26T01:09:03.359 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.359 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.359 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.359 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.305934+0000", 2024-04-26T01:09:03.359 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022386800000000001 2024-04-26T01:09:03.359 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.359 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.359 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.359 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.307007+0000", 2024-04-26T01:09:03.360 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010735930000000001 2024-04-26T01:09:03.360 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.360 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.360 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.360 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.313953+0000", 2024-04-26T01:09:03.360 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006945657 2024-04-26T01:09:03.360 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.360 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.360 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:09:03.360 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316089+0000", 2024-04-26T01:09:03.360 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021360509999999999 2024-04-26T01:09:03.360 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.360 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.360 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.361 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.316313+0000", 2024-04-26T01:09:03.361 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000223558 2024-04-26T01:09:03.361 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.361 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.361 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.361 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.361 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.361 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:6 3.ds0 3:bcd9cb81:::smithi16428812-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:09:03.361 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.303162+0000", 2024-04-26T01:09:03.361 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.044298854000004, 2024-04-26T01:09:03.361 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.027157324, 2024-04-26T01:09:03.361 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.361 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.361 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:09:03.361 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.362 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.362 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.362 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 6 2024-04-26T01:09:03.362 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.362 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.362 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.362 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.362 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303162+0000", 2024-04-26T01:09:03.362 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.362 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.362 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.362 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.362 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303162+0000", 2024-04-26T01:09:03.363 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.363 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.363 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.363 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.363 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303163+0000", 2024-04-26T01:09:03.363 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.145e-06 2024-04-26T01:09:03.363 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.363 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.363 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.363 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303169+0000", 2024-04-26T01:09:03.363 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8150000000000002e-06 2024-04-26T01:09:03.363 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.363 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.363 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.363 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303184+0000", 2024-04-26T01:09:03.366 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5116000000000001e-05 2024-04-26T01:09:03.366 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.366 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.366 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.366 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.303196+0000", 2024-04-26T01:09:03.366 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1778e-05 2024-04-26T01:09:03.366 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.366 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.366 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.366 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308497+0000", 2024-04-26T01:09:03.366 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0053009449999999996 2024-04-26T01:09:03.366 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.366 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.367 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:03.367 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.308937+0000", 2024-04-26T01:09:03.367 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00044013100000000002 2024-04-26T01:09:03.367 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.367 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.367 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.367 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.317636+0000", 2024-04-26T01:09:03.367 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0086997570000000007 2024-04-26T01:09:03.367 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.367 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.367 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.367 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.317809+0000", 2024-04-26T01:09:03.367 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017204899999999999 2024-04-26T01:09:03.368 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.368 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.368 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.368 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.330319+0000", 2024-04-26T01:09:03.368 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012510588 2024-04-26T01:09:03.368 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.368 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.368 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.368 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.368 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.368 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:34 3.ds0 3:b1366e28:::smithi16428812-12:head [append 0~196608,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:03.368 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321139+0000", 2024-04-26T01:09:03.368 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.026321616000004, 2024-04-26T01:09:03.368 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015400954999999999, 2024-04-26T01:09:03.369 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.369 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.369 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:03.369 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.369 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.369 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.369 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 34 2024-04-26T01:09:03.369 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.369 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.369 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.369 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.369 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321139+0000", 2024-04-26T01:09:03.369 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.370 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.370 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.370 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.370 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321139+0000", 2024-04-26T01:09:03.370 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.370 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.370 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.370 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.370 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321142+0000", 2024-04-26T01:09:03.370 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8660000000000002e-06 2024-04-26T01:09:03.370 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.370 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.370 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.371 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321475+0000", 2024-04-26T01:09:03.371 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00033334599999999999 2024-04-26T01:09:03.371 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.371 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.371 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.371 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321504+0000", 2024-04-26T01:09:03.371 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8445e-05 2024-04-26T01:09:03.371 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.371 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.371 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.371 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321518+0000", 2024-04-26T01:09:03.371 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4708e-05 2024-04-26T01:09:03.371 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.371 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.372 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.372 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322532+0000", 2024-04-26T01:09:03.372 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001013246 2024-04-26T01:09:03.372 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.372 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.372 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.372 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322688+0000", 2024-04-26T01:09:03.372 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000156525 2024-04-26T01:09:03.372 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.372 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.372 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.372 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.323904+0000", 2024-04-26T01:09:03.372 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012160160000000001 2024-04-26T01:09:03.373 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.373 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.373 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.373 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.332759+0000", 2024-04-26T01:09:03.373 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0088551199999999993 2024-04-26T01:09:03.373 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.373 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.373 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:09:03.373 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336395+0000", 2024-04-26T01:09:03.373 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0036352559999999999 2024-04-26T01:09:03.373 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.373 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.373 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.373 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336540+0000", 2024-04-26T01:09:03.373 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000145427 2024-04-26T01:09:03.374 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.374 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.374 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.374 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.374 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.374 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:35 3.ds0 3:b1366e28:::smithi16428812-12:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:03.374 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321544+0000", 2024-04-26T01:09:03.374 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.025916498000001, 2024-04-26T01:09:03.374 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015414796, 2024-04-26T01:09:03.374 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.374 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.374 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:03.374 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.374 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.375 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.375 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 35 2024-04-26T01:09:03.375 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.375 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.375 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.375 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.375 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321544+0000", 2024-04-26T01:09:03.375 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.375 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.375 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.375 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.375 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321544+0000", 2024-04-26T01:09:03.375 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.375 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.376 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.376 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.376 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321545+0000", 2024-04-26T01:09:03.376 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3340000000000001e-06 2024-04-26T01:09:03.376 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.376 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.376 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.376 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321553+0000", 2024-04-26T01:09:03.376 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4739999999999999e-06 2024-04-26T01:09:03.376 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.376 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.376 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.376 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321571+0000", 2024-04-26T01:09:03.376 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7688000000000001e-05 2024-04-26T01:09:03.377 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.377 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.377 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.377 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321614+0000", 2024-04-26T01:09:03.377 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3047000000000003e-05 2024-04-26T01:09:03.377 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.377 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.377 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.377 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.324961+0000", 2024-04-26T01:09:03.377 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003347184 2024-04-26T01:09:03.377 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.377 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.377 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.377 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.325113+0000", 2024-04-26T01:09:03.377 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000152287 2024-04-26T01:09:03.378 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.378 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.378 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.378 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.327055+0000", 2024-04-26T01:09:03.378 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001942292 2024-04-26T01:09:03.378 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.378 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.378 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.378 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.332763+0000", 2024-04-26T01:09:03.378 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0057070819999999996 2024-04-26T01:09:03.378 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.378 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.378 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:09:03.378 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336824+0000", 2024-04-26T01:09:03.379 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0040615449999999997 2024-04-26T01:09:03.379 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.379 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.379 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.379 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336959+0000", 2024-04-26T01:09:03.379 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013486300000000001 2024-04-26T01:09:03.379 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.379 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.379 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.379 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.379 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.379 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:36 3.ds0 3:b1366e28:::smithi16428812-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:03.379 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321633+0000", 2024-04-26T01:09:03.379 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.025827660999994, 2024-04-26T01:09:03.380 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.019982366000000001, 2024-04-26T01:09:03.380 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.380 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.380 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:09:03.380 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.380 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.380 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.380 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 36 2024-04-26T01:09:03.380 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.380 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.380 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.380 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.380 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321633+0000", 2024-04-26T01:09:03.381 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.381 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.381 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.381 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.381 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321633+0000", 2024-04-26T01:09:03.381 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.381 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.381 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.381 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.381 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321634+0000", 2024-04-26T01:09:03.381 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.268e-06 2024-04-26T01:09:03.381 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.381 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.381 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.381 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321641+0000", 2024-04-26T01:09:03.382 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5100000000000004e-06 2024-04-26T01:09:03.382 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.382 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.382 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.382 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321656+0000", 2024-04-26T01:09:03.382 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5191000000000001e-05 2024-04-26T01:09:03.382 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.382 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.382 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.382 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321666+0000", 2024-04-26T01:09:03.382 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0178e-05 2024-04-26T01:09:03.382 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.382 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.383 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.383 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.333161+0000", 2024-04-26T01:09:03.383 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011494605999999999 2024-04-26T01:09:03.383 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.383 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.383 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:03.383 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.333311+0000", 2024-04-26T01:09:03.383 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015046100000000001 2024-04-26T01:09:03.383 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.383 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.383 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.383 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338017+0000", 2024-04-26T01:09:03.383 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0047055489999999998 2024-04-26T01:09:03.383 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.383 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.384 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.384 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338172+0000", 2024-04-26T01:09:03.384 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015476500000000001 2024-04-26T01:09:03.384 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.384 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.384 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.384 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341615+0000", 2024-04-26T01:09:03.384 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0034438379999999998 2024-04-26T01:09:03.384 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.384 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.384 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.384 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.384 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.384 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpReadReply(3.ds0 33/30 ECSubReadReply(tid=3, attrs_read=0))", 2024-04-26T01:09:03.385 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.322592+0000", 2024-04-26T01:09:03.385 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.024868650000002, 2024-04-26T01:09:03.385 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077557909999999997, 2024-04-26T01:09:03.385 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.385 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.385 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:09:03.385 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.385 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.385 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.385 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322592+0000", 2024-04-26T01:09:03.385 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.385 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.385 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.385 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.386 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322592+0000", 2024-04-26T01:09:03.386 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.386 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.386 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.386 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322594+0000", 2024-04-26T01:09:03.386 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7740000000000001e-06 2024-04-26T01:09:03.386 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.386 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.386 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322617+0000", 2024-04-26T01:09:03.386 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3074999999999998e-05 2024-04-26T01:09:03.386 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.387 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.387 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322695+0000", 2024-04-26T01:09:03.387 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7892999999999994e-05 2024-04-26T01:09:03.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.387 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.387 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.387 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322710+0000", 2024-04-26T01:09:03.387 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.505e-05 2024-04-26T01:09:03.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.387 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.387 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.387 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329473+0000", 2024-04-26T01:09:03.387 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006763682 2024-04-26T01:09:03.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.387 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.388 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.388 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.330348+0000", 2024-04-26T01:09:03.388 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000874317 2024-04-26T01:09:03.388 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.388 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.388 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.388 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.388 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.388 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 32/30 ECSubWrite(tid=0, reqid=client.4521.0:28, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:09:03.388 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.326088+0000", 2024-04-26T01:09:03.388 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.021372815000007, 2024-04-26T01:09:03.388 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0097951900000000005, 2024-04-26T01:09:03.388 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.388 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.389 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:09:03.389 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.389 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.389 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.389 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326088+0000", 2024-04-26T01:09:03.389 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.389 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.389 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.389 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.389 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326088+0000", 2024-04-26T01:09:03.389 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.389 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.389 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.389 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.390 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326089+0000", 2024-04-26T01:09:03.390 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5460000000000001e-06 2024-04-26T01:09:03.390 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.390 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.390 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.390 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326154+0000", 2024-04-26T01:09:03.390 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4979000000000002e-05 2024-04-26T01:09:03.390 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.390 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.390 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.390 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326251+0000", 2024-04-26T01:09:03.390 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7139999999999995e-05 2024-04-26T01:09:03.390 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.391 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.391 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.391 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.326266+0000", 2024-04-26T01:09:03.391 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4355999999999999e-05 2024-04-26T01:09:03.391 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.391 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.391 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.391 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.330888+0000", 2024-04-26T01:09:03.391 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0046226310000000003 2024-04-26T01:09:03.391 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.391 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.391 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.391 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.331237+0000", 2024-04-26T01:09:03.391 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000348103 2024-04-26T01:09:03.391 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.392 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.392 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.392 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335801+0000", 2024-04-26T01:09:03.392 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0045642189999999996 2024-04-26T01:09:03.392 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.392 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.392 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.392 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335883+0000", 2024-04-26T01:09:03.392 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2216e-05 2024-04-26T01:09:03.392 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.393 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.393 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.393 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.393 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.393 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:46 3.ds0 3:b85e6c81:::smithi16428812-16:head [append 0~262144,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:03.393 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.338039+0000", 2024-04-26T01:09:03.393 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.009421727000003, 2024-04-26T01:09:03.393 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0092704039999999995, 2024-04-26T01:09:03.393 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.393 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.393 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:03.393 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.393 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.394 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.394 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 46 2024-04-26T01:09:03.394 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.394 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.394 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.394 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.394 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338039+0000", 2024-04-26T01:09:03.395 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.395 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.395 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.395 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.395 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338039+0000", 2024-04-26T01:09:03.395 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.395 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.395 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.395 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.395 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338041+0000", 2024-04-26T01:09:03.395 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4660000000000002e-06 2024-04-26T01:09:03.395 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.395 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.396 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.396 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338148+0000", 2024-04-26T01:09:03.396 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000106384 2024-04-26T01:09:03.396 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.396 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.396 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.396 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338170+0000", 2024-04-26T01:09:03.396 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2022e-05 2024-04-26T01:09:03.396 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.396 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.396 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.396 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338184+0000", 2024-04-26T01:09:03.396 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4345000000000001e-05 2024-04-26T01:09:03.396 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.397 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.397 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.397 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341687+0000", 2024-04-26T01:09:03.397 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003502631 2024-04-26T01:09:03.397 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.397 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.397 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.397 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.341929+0000", 2024-04-26T01:09:03.397 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024248 2024-04-26T01:09:03.397 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.397 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.397 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.397 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.343657+0000", 2024-04-26T01:09:03.398 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001727302 2024-04-26T01:09:03.398 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.398 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.398 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.398 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346591+0000", 2024-04-26T01:09:03.398 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0029344420000000002 2024-04-26T01:09:03.398 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.398 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.398 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:09:03.398 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.347148+0000", 2024-04-26T01:09:03.398 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000556801 2024-04-26T01:09:03.398 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.398 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.399 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.399 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.347309+0000", 2024-04-26T01:09:03.399 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000161531 2024-04-26T01:09:03.399 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.399 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.399 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.399 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.399 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.399 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:47 3.ds0 3:b85e6c81:::smithi16428812-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:03.399 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.338373+0000", 2024-04-26T01:09:03.399 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.009088044999999, 2024-04-26T01:09:03.399 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012969223, 2024-04-26T01:09:03.399 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.399 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.400 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:03.400 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.400 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.400 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.400 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 47 2024-04-26T01:09:03.400 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.400 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.400 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.400 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.400 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338373+0000", 2024-04-26T01:09:03.400 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.400 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.400 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.400 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.401 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338373+0000", 2024-04-26T01:09:03.401 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.401 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.401 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.401 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.401 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338374+0000", 2024-04-26T01:09:03.401 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.782e-06 2024-04-26T01:09:03.401 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.401 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.401 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.401 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338383+0000", 2024-04-26T01:09:03.401 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.6570000000000003e-06 2024-04-26T01:09:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338402+0000", 2024-04-26T01:09:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8708000000000001e-05 2024-04-26T01:09:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338415+0000", 2024-04-26T01:09:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2976000000000001e-05 2024-04-26T01:09:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.344766+0000", 2024-04-26T01:09:03.403 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0063516110000000001 2024-04-26T01:09:03.403 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.403 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.403 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.403 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.344969+0000", 2024-04-26T01:09:03.403 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00020233 2024-04-26T01:09:03.403 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.403 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.403 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.403 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.345881+0000", 2024-04-26T01:09:03.403 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00091244999999999998 2024-04-26T01:09:03.403 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.403 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.404 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.404 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.347458+0000", 2024-04-26T01:09:03.404 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001576945 2024-04-26T01:09:03.404 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.404 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:09:03.404 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.351176+0000", 2024-04-26T01:09:03.404 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0037184449999999999 2024-04-26T01:09:03.404 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.404 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.405 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.351342+0000", 2024-04-26T01:09:03.405 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016531900000000001 2024-04-26T01:09:03.405 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.405 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.405 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.405 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.405 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.405 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:48 3.ds0 3:b85e6c81:::smithi16428812-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:03.405 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.338562+0000", 2024-04-26T01:09:03.405 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.008898514999998, 2024-04-26T01:09:03.405 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.028604862000000002, 2024-04-26T01:09:03.406 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.406 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.406 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:09:03.406 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.406 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.406 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.406 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 48 2024-04-26T01:09:03.406 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.406 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.406 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.406 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.407 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338562+0000", 2024-04-26T01:09:03.407 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.407 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.407 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.407 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.407 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338562+0000", 2024-04-26T01:09:03.407 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.407 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.407 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.407 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.407 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338564+0000", 2024-04-26T01:09:03.407 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5319999999999999e-06 2024-04-26T01:09:03.407 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.408 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.408 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338572+0000", 2024-04-26T01:09:03.408 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1780000000000003e-06 2024-04-26T01:09:03.408 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.408 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.408 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338587+0000", 2024-04-26T01:09:03.408 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5293000000000001e-05 2024-04-26T01:09:03.408 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.408 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.408 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338599+0000", 2024-04-26T01:09:03.408 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1968e-05 2024-04-26T01:09:03.409 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.409 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.409 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.409 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346313+0000", 2024-04-26T01:09:03.409 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077139060000000004 2024-04-26T01:09:03.409 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.409 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.409 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:03.409 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346451+0000", 2024-04-26T01:09:03.409 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013819400000000001 2024-04-26T01:09:03.409 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.409 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.409 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.410 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.352103+0000", 2024-04-26T01:09:03.410 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056514249999999999 2024-04-26T01:09:03.410 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.410 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.410 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.410 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.352448+0000", 2024-04-26T01:09:03.410 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034553299999999997 2024-04-26T01:09:03.410 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.410 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.411 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.411 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.367167+0000", 2024-04-26T01:09:03.411 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.014718833000000001 2024-04-26T01:09:03.411 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.411 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.411 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.411 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.411 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.411 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/30 ECSubWrite(tid=2, reqid=client.4521.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=32'2))", 2024-04-26T01:09:03.411 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.339308+0000", 2024-04-26T01:09:03.411 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.008152371999998, 2024-04-26T01:09:03.411 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0072084560000000002, 2024-04-26T01:09:03.411 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.411 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.412 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:09:03.412 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.412 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.412 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.412 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339308+0000", 2024-04-26T01:09:03.412 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.412 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.412 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.412 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.412 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339308+0000", 2024-04-26T01:09:03.412 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.412 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.412 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.413 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.413 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339310+0000", 2024-04-26T01:09:03.413 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4959999999999999e-06 2024-04-26T01:09:03.413 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.413 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.413 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.413 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339328+0000", 2024-04-26T01:09:03.413 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8349999999999999e-05 2024-04-26T01:09:03.413 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.413 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.413 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.413 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339614+0000", 2024-04-26T01:09:03.413 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028623 2024-04-26T01:09:03.413 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.414 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.414 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.414 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339629+0000", 2024-04-26T01:09:03.414 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4803999999999999e-05 2024-04-26T01:09:03.414 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.414 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.414 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.414 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.340313+0000", 2024-04-26T01:09:03.414 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00068382200000000005 2024-04-26T01:09:03.415 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.415 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.415 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.415 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.340400+0000", 2024-04-26T01:09:03.415 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.7321e-05 2024-04-26T01:09:03.415 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.415 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.415 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.415 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346462+0000", 2024-04-26T01:09:03.415 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0060617939999999997 2024-04-26T01:09:03.415 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.415 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.415 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.416 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.346517+0000", 2024-04-26T01:09:03.416 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4639000000000003e-05 2024-04-26T01:09:03.416 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.416 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.416 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.416 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/30 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=38'19, trim_to=0'0, roll_forward_to=37'17))", 2024-04-26T01:09:03.416 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:04.427965+0000", 2024-04-26T01:09:03.416 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 58.919495828000002, 2024-04-26T01:09:03.416 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0080739560000000002, 2024-04-26T01:09:03.417 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.417 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.417 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:09:03.417 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.417 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.417 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.427965+0000", 2024-04-26T01:09:03.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.417 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.417 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.427965+0000", 2024-04-26T01:09:03.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.418 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.418 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.427966+0000", 2024-04-26T01:09:03.418 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3600000000000003e-07 2024-04-26T01:09:03.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.418 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.418 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.428022+0000", 2024-04-26T01:09:03.418 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6805999999999998e-05 2024-04-26T01:09:03.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.418 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.418 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.428186+0000", 2024-04-26T01:09:03.418 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000163287 2024-04-26T01:09:03.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.419 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.419 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.428212+0000", 2024-04-26T01:09:03.419 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6163e-05 2024-04-26T01:09:03.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.419 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.419 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.430074+0000", 2024-04-26T01:09:03.419 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0018619369999999999 2024-04-26T01:09:03.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.419 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.419 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.430098+0000", 2024-04-26T01:09:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.393e-05 2024-04-26T01:09:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.435960+0000", 2024-04-26T01:09:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0058619930000000002 2024-04-26T01:09:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:04.436039+0000", 2024-04-26T01:09:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9103999999999993e-05 2024-04-26T01:09:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.421 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 60/30 ECSubWrite(tid=51, reqid=osd.15.0:51, at_version=60'45, trim_to=0'0, roll_forward_to=59'43))", 2024-04-26T01:09:03.421 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:09:03.421 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 37.175461026000001, 2024-04-26T01:09:03.421 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.15434805400000001, 2024-04-26T01:09:03.421 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.421 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.421 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:09:03.421 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.421 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.421 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:09:03.421 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.422 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.422 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:09:03.422 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.422 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.422 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172000+0000", 2024-04-26T01:09:03.422 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7800000000000001e-07 2024-04-26T01:09:03.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.422 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.422 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.172035+0000", 2024-04-26T01:09:03.422 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.5151999999999998e-05 2024-04-26T01:09:03.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.423 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.423 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324571+0000", 2024-04-26T01:09:03.423 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.152535484 2024-04-26T01:09:03.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.423 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.423 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324589+0000", 2024-04-26T01:09:03.423 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8323999999999999e-05 2024-04-26T01:09:03.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.423 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.423 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324739+0000", 2024-04-26T01:09:03.423 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014940099999999999 2024-04-26T01:09:03.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.424 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.424 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.324758+0000", 2024-04-26T01:09:03.424 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9562999999999999e-05 2024-04-26T01:09:03.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.424 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.424 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.326281+0000", 2024-04-26T01:09:03.424 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001523349 2024-04-26T01:09:03.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.424 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.425 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.326348+0000", 2024-04-26T01:09:03.425 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6203000000000003e-05 2024-04-26T01:09:03.425 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.425 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.425 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.425 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.425 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.bs5 60/30 ECSubWrite(tid=349, reqid=osd.4.0:349, at_version=60'52, trim_to=0'0, roll_forward_to=59'50))", 2024-04-26T01:09:03.425 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:09:03.425 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 37.169884128, 2024-04-26T01:09:03.425 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.007081085, 2024-04-26T01:09:03.425 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.425 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.425 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:09:03.426 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.426 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.426 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:09:03.426 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.426 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.426 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.426 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:09:03.426 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.426 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.426 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177577+0000", 2024-04-26T01:09:03.427 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1799999999999999e-07 2024-04-26T01:09:03.427 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177638+0000", 2024-04-26T01:09:03.427 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1736000000000007e-05 2024-04-26T01:09:03.427 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177710+0000", 2024-04-26T01:09:03.427 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1756999999999998e-05 2024-04-26T01:09:03.427 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.428 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.428 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177722+0000", 2024-04-26T01:09:03.428 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1516e-05 2024-04-26T01:09:03.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.428 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.428 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.428 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177868+0000", 2024-04-26T01:09:03.428 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000146112 2024-04-26T01:09:03.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.428 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.428 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.428 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.177896+0000", 2024-04-26T01:09:03.428 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7943999999999999e-05 2024-04-26T01:09:03.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.429 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.429 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.184603+0000", 2024-04-26T01:09:03.429 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006707691 2024-04-26T01:09:03.429 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.429 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.429 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:26.184658+0000", 2024-04-26T01:09:03.429 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4110999999999997e-05 2024-04-26T01:09:03.429 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.429 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.429 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.429 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.430 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:780 3.ds0 3:bcd9cb81:::smithi16428812-2:head [delete] snapc 15=[15] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:09:03.430 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:38.244242+0000", 2024-04-26T01:09:03.430 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 25.103218908999999, 2024-04-26T01:09:03.430 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0084976549999999998, 2024-04-26T01:09:03.430 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.430 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.430 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:03.430 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.430 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.430 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.430 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 780 2024-04-26T01:09:03.430 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.430 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.430 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.431 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.431 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244242+0000", 2024-04-26T01:09:03.431 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.431 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.431 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.431 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244242+0000", 2024-04-26T01:09:03.431 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.431 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.431 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.431 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244246+0000", 2024-04-26T01:09:03.431 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8260000000000003e-06 2024-04-26T01:09:03.431 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244264+0000", 2024-04-26T01:09:03.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8209000000000001e-05 2024-04-26T01:09:03.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244294+0000", 2024-04-26T01:09:03.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0099000000000001e-05 2024-04-26T01:09:03.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244310+0000", 2024-04-26T01:09:03.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6336000000000001e-05 2024-04-26T01:09:03.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244466+0000", 2024-04-26T01:09:03.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015627100000000001 2024-04-26T01:09:03.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2024-04-26T01:09:03.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.244486+0000", 2024-04-26T01:09:03.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9678999999999999e-05 2024-04-26T01:09:03.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.246633+0000", 2024-04-26T01:09:03.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021467359999999998 2024-04-26T01:09:03.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.434 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.434 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.434 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.246783+0000", 2024-04-26T01:09:03.434 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015011500000000001 2024-04-26T01:09:03.434 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.434 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.434 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.434 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.248075+0000", 2024-04-26T01:09:03.434 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012918890000000001 2024-04-26T01:09:03.434 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.434 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.434 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.434 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.249771+0000", 2024-04-26T01:09:03.435 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0016958209999999999 2024-04-26T01:09:03.435 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.435 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.435 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:09:03.435 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.252619+0000", 2024-04-26T01:09:03.435 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002848214 2024-04-26T01:09:03.435 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.435 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.435 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.435 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.252739+0000", 2024-04-26T01:09:03.435 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012046 2024-04-26T01:09:03.435 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.435 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.435 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.436 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.3s1 72/30 ECSubWrite(tid=240, reqid=client.4521.0:781, at_version=72'20, trim_to=0'0, roll_forward_to=63'18))", 2024-04-26T01:09:03.436 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:38.263581+0000", 2024-04-26T01:09:03.436 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 25.083879340999999, 2024-04-26T01:09:03.436 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0089752990000000008, 2024-04-26T01:09:03.436 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.436 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.436 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:09:03.436 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.436 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.263581+0000", 2024-04-26T01:09:03.436 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.437 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.263581+0000", 2024-04-26T01:09:03.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.437 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.263583+0000", 2024-04-26T01:09:03.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.39e-06 2024-04-26T01:09:03.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.438 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.263949+0000", 2024-04-26T01:09:03.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00036607399999999998 2024-04-26T01:09:03.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.438 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.264035+0000", 2024-04-26T01:09:03.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.6055000000000002e-05 2024-04-26T01:09:03.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.438 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.264059+0000", 2024-04-26T01:09:03.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4071999999999999e-05 2024-04-26T01:09:03.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.439 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.439 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.264254+0000", 2024-04-26T01:09:03.439 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019476199999999999 2024-04-26T01:09:03.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.439 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.439 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.264282+0000", 2024-04-26T01:09:03.439 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8768999999999998e-05 2024-04-26T01:09:03.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.439 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.439 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.272454+0000", 2024-04-26T01:09:03.440 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0081717030000000006 2024-04-26T01:09:03.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.440 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.440 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.272557+0000", 2024-04-26T01:09:03.440 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010247399999999999 2024-04-26T01:09:03.440 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.440 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.440 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.440 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 72/30 ECSubWrite(tid=56, reqid=client.4521.0:785, at_version=72'49, trim_to=0'0, roll_forward_to=72'47))", 2024-04-26T01:09:03.440 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:38.303386+0000", 2024-04-26T01:09:03.440 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 25.044074988999999, 2024-04-26T01:09:03.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0094813009999999993, 2024-04-26T01:09:03.441 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.441 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.441 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:09:03.441 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.441 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.441 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.303386+0000", 2024-04-26T01:09:03.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.441 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.441 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.303386+0000", 2024-04-26T01:09:03.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.442 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.442 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.303388+0000", 2024-04-26T01:09:03.442 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8439999999999998e-06 2024-04-26T01:09:03.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.442 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.442 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.304973+0000", 2024-04-26T01:09:03.442 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0015846090000000001 2024-04-26T01:09:03.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.443 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.443 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.305084+0000", 2024-04-26T01:09:03.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011120099999999999 2024-04-26T01:09:03.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.443 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.443 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.305116+0000", 2024-04-26T01:09:03.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1341999999999999e-05 2024-04-26T01:09:03.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.443 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.443 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.305312+0000", 2024-04-26T01:09:03.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000196286 2024-04-26T01:09:03.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.444 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.305347+0000", 2024-04-26T01:09:03.444 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.5185000000000003e-05 2024-04-26T01:09:03.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.444 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.312795+0000", 2024-04-26T01:09:03.444 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0074474119999999996 2024-04-26T01:09:03.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.444 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.312867+0000", 2024-04-26T01:09:03.444 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2422000000000006e-05 2024-04-26T01:09:03.444 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.445 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.445 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.445 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.5s4 81/30 ECSubWrite(tid=67, reqid=osd.15.0:67, at_version=81'54, trim_to=0'0, roll_forward_to=74'50))", 2024-04-26T01:09:03.445 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:47.314584+0000", 2024-04-26T01:09:03.445 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 16.032876373000001, 2024-04-26T01:09:03.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0072634040000000002, 2024-04-26T01:09:03.445 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.445 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.445 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:09:03.445 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.445 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.446 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314584+0000", 2024-04-26T01:09:03.446 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.446 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.446 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314584+0000", 2024-04-26T01:09:03.446 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.446 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.446 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314586+0000", 2024-04-26T01:09:03.446 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2729999999999999e-06 2024-04-26T01:09:03.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314621+0000", 2024-04-26T01:09:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.5129999999999997e-05 2024-04-26T01:09:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314685+0000", 2024-04-26T01:09:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4212000000000001e-05 2024-04-26T01:09:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.314699+0000", 2024-04-26T01:09:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4453e-05 2024-04-26T01:09:03.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.448 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.316889+0000", 2024-04-26T01:09:03.448 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002189954 2024-04-26T01:09:03.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.448 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.316916+0000", 2024-04-26T01:09:03.448 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6441999999999999e-05 2024-04-26T01:09:03.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.448 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.321756+0000", 2024-04-26T01:09:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0048401130000000001 2024-04-26T01:09:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:47.321848+0000", 2024-04-26T01:09:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1827e-05 2024-04-26T01:09:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(osd.4.21:16 3.ds0 3:b85e6c81:::smithi16428812-16:head [assert-version v15,copy-get max 8388608] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-04-26T01:09:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:48.340027+0000", 2024-04-26T01:09:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 15.007433749, 2024-04-26T01:09:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0093623700000000001, 2024-04-26T01:09:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:09:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.340027+0000", 2024-04-26T01:09:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.340027+0000", 2024-04-26T01:09:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.340028+0000", 2024-04-26T01:09:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1200000000000001e-07 2024-04-26T01:09:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.340033+0000", 2024-04-26T01:09:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.3199999999999999e-06 2024-04-26T01:09:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.340046+0000", 2024-04-26T01:09:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2992e-05 2024-04-26T01:09:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.340054+0000", 2024-04-26T01:09:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.5020000000000003e-06 2024-04-26T01:09:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.341775+0000", 2024-04-26T01:09:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001721045 2024-04-26T01:09:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.342038+0000", 2024-04-26T01:09:03.453 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000263527 2024-04-26T01:09:03.453 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.453 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.453 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.453 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:48.349389+0000", 2024-04-26T01:09:03.453 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0073510720000000002 2024-04-26T01:09:03.453 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.453 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.453 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.453 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.453 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.453 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:09:03.455 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:09:03.583 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:09:03.583 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:09:03.583 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:09:03.583 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:09:03.598 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:09:03.721 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:09:03.721 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:09:03.721 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2024-04-26T01:09:03.721 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2024-04-26T01:09:03.722 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:09:03.735 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:09:03.861 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:09:03.862 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2024-04-26T01:09:03.862 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2024-04-26T01:09:03.862 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2024-04-26T01:09:03.862 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.862 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:29 3.5s0 3:acf4d671:::smithi16428812-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:09:03.862 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.321678+0000", 2024-04-26T01:09:03.862 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.538499719000001, 2024-04-26T01:09:03.862 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.016320530999999999, 2024-04-26T01:09:03.862 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.862 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.862 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:03.863 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.863 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.863 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.863 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 29 2024-04-26T01:09:03.863 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.863 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.863 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.863 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.863 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321678+0000", 2024-04-26T01:09:03.863 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.863 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.863 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.863 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.864 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321678+0000", 2024-04-26T01:09:03.864 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.864 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.864 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.864 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.864 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321680+0000", 2024-04-26T01:09:03.864 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1680000000000001e-06 2024-04-26T01:09:03.864 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.864 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.864 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.864 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321686+0000", 2024-04-26T01:09:03.864 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9539999999999998e-06 2024-04-26T01:09:03.865 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.865 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.865 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.865 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321702+0000", 2024-04-26T01:09:03.865 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5241e-05 2024-04-26T01:09:03.865 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.865 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.865 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.865 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.321716+0000", 2024-04-26T01:09:03.865 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4384e-05 2024-04-26T01:09:03.865 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.865 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.865 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.866 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329428+0000", 2024-04-26T01:09:03.866 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077114369999999998 2024-04-26T01:09:03.866 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.866 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.866 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.866 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.329669+0000", 2024-04-26T01:09:03.866 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000241561 2024-04-26T01:09:03.866 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.866 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.866 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.866 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.331103+0000", 2024-04-26T01:09:03.866 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001433871 2024-04-26T01:09:03.866 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.866 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.867 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.867 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.336734+0000", 2024-04-26T01:09:03.867 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.005631306 2024-04-26T01:09:03.867 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.867 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.867 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:09:03.867 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337654+0000", 2024-04-26T01:09:03.867 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000919716 2024-04-26T01:09:03.867 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.867 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.868 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.868 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.337999+0000", 2024-04-26T01:09:03.868 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034489299999999998 2024-04-26T01:09:03.868 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.868 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.868 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.868 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.868 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.868 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:30 3.5s0 3:acf4d671:::smithi16428812-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-26T01:09:03.869 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.322117+0000", 2024-04-26T01:09:03.869 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.538060664, 2024-04-26T01:09:03.869 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.021239319999999999, 2024-04-26T01:09:03.869 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.869 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.869 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:09:03.869 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.869 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.869 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.869 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 30 2024-04-26T01:09:03.869 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.869 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.869 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.869 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.870 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322117+0000", 2024-04-26T01:09:03.870 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.870 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.870 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.870 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.870 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322117+0000", 2024-04-26T01:09:03.870 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.870 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.870 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.870 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.870 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322119+0000", 2024-04-26T01:09:03.870 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3740000000000001e-06 2024-04-26T01:09:03.870 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.871 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.871 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.871 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322127+0000", 2024-04-26T01:09:03.871 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.994e-06 2024-04-26T01:09:03.871 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.871 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.871 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.871 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322146+0000", 2024-04-26T01:09:03.871 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9205999999999999e-05 2024-04-26T01:09:03.871 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.871 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.871 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.871 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.322159+0000", 2024-04-26T01:09:03.871 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3377000000000001e-05 2024-04-26T01:09:03.872 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.872 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.872 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.872 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.334766+0000", 2024-04-26T01:09:03.872 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012606732000000001 2024-04-26T01:09:03.872 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.872 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.872 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:03.872 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.335000+0000", 2024-04-26T01:09:03.872 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00023435799999999999 2024-04-26T01:09:03.872 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.872 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.872 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.872 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.338932+0000", 2024-04-26T01:09:03.873 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0039317500000000003 2024-04-26T01:09:03.873 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.873 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.873 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.873 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.339070+0000", 2024-04-26T01:09:03.873 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000137315 2024-04-26T01:09:03.873 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.873 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.873 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.873 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.343357+0000", 2024-04-26T01:09:03.873 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042872140000000001 2024-04-26T01:09:03.873 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.873 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.873 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.873 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.874 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.874 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:128 3.5s0 3:a661dd24:::smithi16428812-43:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:03.874 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:09:03.874 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.403692540999998, 2024-04-26T01:09:03.874 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.083673129999999998, 2024-04-26T01:09:03.874 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.874 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.874 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:03.874 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.874 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.874 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.874 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 128 2024-04-26T01:09:03.874 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.875 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.875 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.875 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.875 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:09:03.875 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.875 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.875 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.875 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.875 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:09:03.875 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.875 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.875 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.875 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.875 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456486+0000", 2024-04-26T01:09:03.875 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1299999999999999e-07 2024-04-26T01:09:03.876 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.876 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.876 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.876 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456490+0000", 2024-04-26T01:09:03.876 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4029999999999999e-06 2024-04-26T01:09:03.876 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.876 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.876 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.876 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456501+0000", 2024-04-26T01:09:03.876 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1124999999999999e-05 2024-04-26T01:09:03.876 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.876 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.876 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.876 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456507+0000", 2024-04-26T01:09:03.877 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3180000000000003e-06 2024-04-26T01:09:03.877 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.877 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.877 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.877 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.458775+0000", 2024-04-26T01:09:03.877 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002268061 2024-04-26T01:09:03.877 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.877 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.877 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.877 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.458948+0000", 2024-04-26T01:09:03.877 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017263900000000001 2024-04-26T01:09:03.877 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.877 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.878 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.878 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.459498+0000", 2024-04-26T01:09:03.878 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00055037900000000004 2024-04-26T01:09:03.878 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.878 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.878 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.879 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460166+0000", 2024-04-26T01:09:03.879 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00066757899999999996 2024-04-26T01:09:03.879 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.879 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.879 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:09:03.879 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540012+0000", 2024-04-26T01:09:03.879 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.079846146000000007 2024-04-26T01:09:03.879 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.879 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.879 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.879 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540159+0000", 2024-04-26T01:09:03.880 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014676700000000001 2024-04-26T01:09:03.880 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.880 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.880 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.880 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.880 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.880 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:129 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:03.880 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:09:03.880 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.403663570000006, 2024-04-26T01:09:03.880 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.09570555, 2024-04-26T01:09:03.880 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.880 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.880 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:09:03.880 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.881 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.881 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.881 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 129 2024-04-26T01:09:03.881 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.881 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.881 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.881 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.881 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:09:03.881 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.881 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.881 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.881 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.881 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:09:03.881 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.882 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.882 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.882 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.882 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456515+0000", 2024-04-26T01:09:03.882 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0699999999999997e-07 2024-04-26T01:09:03.882 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.882 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.882 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.882 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456518+0000", 2024-04-26T01:09:03.882 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.898e-06 2024-04-26T01:09:03.882 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.882 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.882 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.882 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456532+0000", 2024-04-26T01:09:03.883 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3597e-05 2024-04-26T01:09:03.883 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.883 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.883 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.883 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.456537+0000", 2024-04-26T01:09:03.883 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.242e-06 2024-04-26T01:09:03.883 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.883 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.883 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.883 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.459839+0000", 2024-04-26T01:09:03.883 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033019830000000001 2024-04-26T01:09:03.883 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.883 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.883 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:03.884 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.459972+0000", 2024-04-26T01:09:03.884 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000133199 2024-04-26T01:09:03.884 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.884 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.884 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.884 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540208+0000", 2024-04-26T01:09:03.884 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.080236319 2024-04-26T01:09:03.884 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.884 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.884 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.884 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540275+0000", 2024-04-26T01:09:03.884 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6486e-05 2024-04-26T01:09:03.884 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.884 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.884 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.885 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.552220+0000", 2024-04-26T01:09:03.885 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011945318999999999 2024-04-26T01:09:03.885 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.885 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.885 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.885 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.885 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.885 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 33/30 ECSubWriteReply(tid=9, last_complete=33'6, committed=1, applied=1))", 2024-04-26T01:09:03.885 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:09:03.885 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.399882496000004, 2024-04-26T01:09:03.885 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.079879818000000005, 2024-04-26T01:09:03.885 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.885 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.886 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:09:03.886 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.886 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.886 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.886 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:09:03.886 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.886 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.886 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.886 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.886 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:09:03.886 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.886 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.886 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.886 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460296+0000", 2024-04-26T01:09:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6599999999999996e-07 2024-04-26T01:09:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.460301+0000", 2024-04-26T01:09:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4689999999999999e-06 2024-04-26T01:09:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.539830+0000", 2024-04-26T01:09:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.079528926999999999 2024-04-26T01:09:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.888 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.539834+0000", 2024-04-26T01:09:03.888 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5430000000000004e-06 2024-04-26T01:09:03.888 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.888 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.888 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.888 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.539887+0000", 2024-04-26T01:09:03.888 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2998999999999999e-05 2024-04-26T01:09:03.888 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.888 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.888 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.888 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.540175+0000", 2024-04-26T01:09:03.888 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028821400000000002 2024-04-26T01:09:03.888 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.889 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.889 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.889 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.889 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.889 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:183 3.5s0 3:a8ed6f44:::smithi16428812-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-04-26T01:09:03.889 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:09:03.889 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.324984508, 2024-04-26T01:09:03.889 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.017578818999999999, 2024-04-26T01:09:03.889 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.889 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.889 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:09:03.889 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.889 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.889 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.889 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 183 2024-04-26T01:09:03.890 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.890 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.890 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.890 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.890 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:09:03.890 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.890 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.890 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.890 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.890 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:09:03.890 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.890 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.890 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.890 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.891 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535194+0000", 2024-04-26T01:09:03.891 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9500000000000002e-07 2024-04-26T01:09:03.891 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.891 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.891 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.891 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535198+0000", 2024-04-26T01:09:03.891 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6430000000000001e-06 2024-04-26T01:09:03.891 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.891 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.891 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.891 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535207+0000", 2024-04-26T01:09:03.891 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9730000000000003e-06 2024-04-26T01:09:03.891 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.892 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.892 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.892 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.535215+0000", 2024-04-26T01:09:03.892 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1540000000000002e-06 2024-04-26T01:09:03.892 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.892 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.892 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.892 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.537794+0000", 2024-04-26T01:09:03.892 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0025788740000000001 2024-04-26T01:09:03.892 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.892 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.892 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:03.892 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.537868+0000", 2024-04-26T01:09:03.892 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4028000000000003e-05 2024-04-26T01:09:03.893 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.893 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.893 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.893 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.550967+0000", 2024-04-26T01:09:03.893 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013099006999999999 2024-04-26T01:09:03.893 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.893 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.893 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.893 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551052+0000", 2024-04-26T01:09:03.893 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4667999999999996e-05 2024-04-26T01:09:03.893 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.893 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.893 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.893 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.552772+0000", 2024-04-26T01:09:03.894 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0017207769999999999 2024-04-26T01:09:03.894 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.894 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.894 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.894 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.894 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.894 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 33/30 ECSubWriteReply(tid=14, last_complete=33'9, committed=1, applied=1))", 2024-04-26T01:09:03.894 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:07:59.551733+0000", 2024-04-26T01:09:03.894 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 64.308445047000006, 2024-04-26T01:09:03.894 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.40949085299999999, 2024-04-26T01:09:03.894 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.894 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.894 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:09:03.894 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.895 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.895 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.895 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551733+0000", 2024-04-26T01:09:03.895 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.895 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.895 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.895 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.895 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551733+0000", 2024-04-26T01:09:03.895 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.895 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.895 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.895 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.895 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551734+0000", 2024-04-26T01:09:03.896 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7199999999999998e-07 2024-04-26T01:09:03.896 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.896 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.896 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.896 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.551739+0000", 2024-04-26T01:09:03.896 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9520000000000002e-06 2024-04-26T01:09:03.896 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.896 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.896 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.896 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.960976+0000", 2024-04-26T01:09:03.896 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.40923691400000001 2024-04-26T01:09:03.896 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.896 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.897 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.897 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.960989+0000", 2024-04-26T01:09:03.897 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3536000000000001e-05 2024-04-26T01:09:03.897 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.897 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.897 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.897 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.961157+0000", 2024-04-26T01:09:03.897 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016794600000000001 2024-04-26T01:09:03.897 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.897 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.897 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.897 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:07:59.961224+0000", 2024-04-26T01:09:03.897 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6733e-05 2024-04-26T01:09:03.897 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.898 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.898 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.898 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.898 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.898 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:251 3.5s0 3:acf4d671:::smithi16428812-10:head [append 0~0] snapc 4=[4,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:09:03.898 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:03.428674+0000", 2024-04-26T01:09:03.898 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 60.43150361, 2024-04-26T01:09:03.898 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.48188827299999998, 2024-04-26T01:09:03.898 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.898 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.898 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:03.898 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.898 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.899 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.899 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 251 2024-04-26T01:09:03.899 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.899 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.899 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.899 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.899 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428674+0000", 2024-04-26T01:09:03.899 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.899 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.899 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.899 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.899 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428674+0000", 2024-04-26T01:09:03.899 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.899 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.899 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.900 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.900 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428676+0000", 2024-04-26T01:09:03.900 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6559999999999999e-06 2024-04-26T01:09:03.900 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.900 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.900 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.900 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428685+0000", 2024-04-26T01:09:03.900 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0869999999999996e-06 2024-04-26T01:09:03.900 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.900 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.900 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.900 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428707+0000", 2024-04-26T01:09:03.900 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1365999999999998e-05 2024-04-26T01:09:03.900 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.901 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.901 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.901 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.428723+0000", 2024-04-26T01:09:03.901 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6563e-05 2024-04-26T01:09:03.901 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.901 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.901 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.901 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.434334+0000", 2024-04-26T01:09:03.901 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056104170000000004 2024-04-26T01:09:03.901 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.901 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.901 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.901 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.434677+0000", 2024-04-26T01:09:03.901 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000343937 2024-04-26T01:09:03.901 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.902 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.902 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.902 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.436169+0000", 2024-04-26T01:09:03.902 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014915939999999999 2024-04-26T01:09:03.902 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.902 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.902 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.902 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.438266+0000", 2024-04-26T01:09:03.902 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002096746 2024-04-26T01:09:03.902 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.902 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.902 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:09:03.902 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.910445+0000", 2024-04-26T01:09:03.902 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.47217926300000002 2024-04-26T01:09:03.903 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.903 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.903 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.903 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:03.910563+0000", 2024-04-26T01:09:03.903 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000117644 2024-04-26T01:09:03.903 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.903 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.903 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.903 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.903 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.903 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:416 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-26T01:09:03.903 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.443980+0000", 2024-04-26T01:09:03.903 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 53.416198219999998, 2024-04-26T01:09:03.903 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.018880322000000001, 2024-04-26T01:09:03.903 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.904 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.904 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:09:03.904 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.904 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.904 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.904 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 416 2024-04-26T01:09:03.904 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.904 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.904 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.904 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.904 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443980+0000", 2024-04-26T01:09:03.904 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.904 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.904 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.904 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.905 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443980+0000", 2024-04-26T01:09:03.905 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.905 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.905 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.905 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.905 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443981+0000", 2024-04-26T01:09:03.905 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7000000000000003e-07 2024-04-26T01:09:03.905 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.905 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.905 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.905 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443984+0000", 2024-04-26T01:09:03.905 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.36e-06 2024-04-26T01:09:03.905 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.906 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.906 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.906 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.443994+0000", 2024-04-26T01:09:03.906 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0285999999999999e-05 2024-04-26T01:09:03.906 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.906 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.906 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.906 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.444002+0000", 2024-04-26T01:09:03.906 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4970000000000004e-06 2024-04-26T01:09:03.906 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.906 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.906 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.906 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.454900+0000", 2024-04-26T01:09:03.906 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010897784000000001 2024-04-26T01:09:03.907 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.907 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.907 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2024-04-26T01:09:03.907 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.455102+0000", 2024-04-26T01:09:03.907 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00020183000000000001 2024-04-26T01:09:03.907 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.907 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.907 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.907 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.460460+0000", 2024-04-26T01:09:03.907 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0053580140000000004 2024-04-26T01:09:03.907 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.907 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.907 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.907 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.460760+0000", 2024-04-26T01:09:03.907 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00030007900000000002 2024-04-26T01:09:03.908 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.908 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.908 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.908 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.462860+0000", 2024-04-26T01:09:03.908 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021005020000000002 2024-04-26T01:09:03.908 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.908 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.908 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.908 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.908 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.908 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.3s3 44/30 ECSubWrite(tid=198, reqid=client.4521.0:420, at_version=44'13, trim_to=0'0, roll_forward_to=44'11))", 2024-04-26T01:09:03.908 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:10.466337+0000", 2024-04-26T01:09:03.908 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 53.393841463000001, 2024-04-26T01:09:03.908 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.57864738100000002, 2024-04-26T01:09:03.909 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.909 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.909 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:09:03.909 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.909 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.909 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.909 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466337+0000", 2024-04-26T01:09:03.909 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.909 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.909 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.909 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.909 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466337+0000", 2024-04-26T01:09:03.909 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.909 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.910 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.910 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.910 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466338+0000", 2024-04-26T01:09:03.910 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4479999999999999e-06 2024-04-26T01:09:03.910 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.910 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.910 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.910 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:10.466374+0000", 2024-04-26T01:09:03.910 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6137999999999999e-05 2024-04-26T01:09:03.910 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.910 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.910 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.910 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.043926+0000", 2024-04-26T01:09:03.910 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.57755188800000001 2024-04-26T01:09:03.910 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.911 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.911 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.911 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.043944+0000", 2024-04-26T01:09:03.911 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7745999999999999e-05 2024-04-26T01:09:03.911 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.911 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.911 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.911 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044049+0000", 2024-04-26T01:09:03.911 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010500700000000001 2024-04-26T01:09:03.911 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.911 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.911 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.911 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044068+0000", 2024-04-26T01:09:03.911 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8984999999999999e-05 2024-04-26T01:09:03.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.912 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.912 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.912 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044949+0000", 2024-04-26T01:09:03.912 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00088138400000000003 2024-04-26T01:09:03.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.912 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.912 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.912 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:11.044984+0000", 2024-04-26T01:09:03.912 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4785e-05 2024-04-26T01:09:03.912 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.912 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.912 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.912 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.913 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:554 3.5s0 3:a661dd24:::smithi16428812-43:head [copy-from ver 26] snapc e=[e,d,b,7] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-04-26T01:09:03.913 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:21.094110+0000", 2024-04-26T01:09:03.913 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 42.766067673999999, 2024-04-26T01:09:03.913 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.050017671, 2024-04-26T01:09:03.913 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.913 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.913 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:03.913 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.913 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.913 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.913 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 554 2024-04-26T01:09:03.913 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.913 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.913 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.914 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.914 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094110+0000", 2024-04-26T01:09:03.914 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.914 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.914 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.914 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.914 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094110+0000", 2024-04-26T01:09:03.914 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.914 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.914 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.914 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.914 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094114+0000", 2024-04-26T01:09:03.914 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.89e-06 2024-04-26T01:09:03.914 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.915 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.915 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.915 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094134+0000", 2024-04-26T01:09:03.915 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0007e-05 2024-04-26T01:09:03.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.915 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.915 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.915 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094181+0000", 2024-04-26T01:09:03.915 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6731999999999999e-05 2024-04-26T01:09:03.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.915 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.915 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.915 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094205+0000", 2024-04-26T01:09:03.915 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4326999999999999e-05 2024-04-26T01:09:03.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.916 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.916 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.916 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094455+0000", 2024-04-26T01:09:03.916 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024929999999999999 2024-04-26T01:09:03.916 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.916 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.916 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2024-04-26T01:09:03.916 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094477+0000", 2024-04-26T01:09:03.916 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2646e-05 2024-04-26T01:09:03.916 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.916 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.916 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.916 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105141+0000", 2024-04-26T01:09:03.916 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010664167 2024-04-26T01:09:03.916 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.917 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.917 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.917 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105392+0000", 2024-04-26T01:09:03.917 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00025070700000000002 2024-04-26T01:09:03.917 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.917 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.917 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.917 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.135272+0000", 2024-04-26T01:09:03.917 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.029880263000000001 2024-04-26T01:09:03.917 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.917 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.917 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.917 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.142569+0000", 2024-04-26T01:09:03.917 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0072964270000000003 2024-04-26T01:09:03.918 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.918 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.918 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:09:03.918 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.142755+0000", 2024-04-26T01:09:03.918 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00018603399999999999 2024-04-26T01:09:03.918 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.918 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.918 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.918 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.144128+0000", 2024-04-26T01:09:03.918 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0013731710000000001 2024-04-26T01:09:03.918 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.918 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.918 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.918 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.918 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.919 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:555 3.5s0 3:a661dd24:::smithi16428812-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e55)", 2024-04-26T01:09:03.919 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:21.094258+0000", 2024-04-26T01:09:03.919 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 42.765920508999997, 2024-04-26T01:09:03.919 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.045831917999999999, 2024-04-26T01:09:03.919 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.919 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.919 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:09:03.919 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.919 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.919 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.919 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 555 2024-04-26T01:09:03.919 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.919 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.920 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.920 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.920 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094258+0000", 2024-04-26T01:09:03.920 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.920 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.920 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.920 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.920 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094258+0000", 2024-04-26T01:09:03.920 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.920 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.920 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.920 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.920 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094259+0000", 2024-04-26T01:09:03.920 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7370000000000001e-06 2024-04-26T01:09:03.920 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.921 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.921 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.921 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094268+0000", 2024-04-26T01:09:03.921 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8359999999999998e-06 2024-04-26T01:09:03.921 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.921 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.921 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.921 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094289+0000", 2024-04-26T01:09:03.921 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0937e-05 2024-04-26T01:09:03.921 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.921 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.921 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.921 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094299+0000", 2024-04-26T01:09:03.922 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.029e-05 2024-04-26T01:09:03.922 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.922 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.922 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.922 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094609+0000", 2024-04-26T01:09:03.922 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00031005900000000001 2024-04-26T01:09:03.922 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.922 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.922 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:09:03.922 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.094619+0000", 2024-04-26T01:09:03.922 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9799999999999993e-06 2024-04-26T01:09:03.922 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.922 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.922 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105868+0000", 2024-04-26T01:09:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011248555 2024-04-26T01:09:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2024-04-26T01:09:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.105946+0000", 2024-04-26T01:09:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8005999999999998e-05 2024-04-26T01:09:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.139672+0000", 2024-04-26T01:09:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.033725805999999997 2024-04-26T01:09:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.924 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.924 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.139869+0000", 2024-04-26T01:09:03.924 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019683 2024-04-26T01:09:03.924 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.924 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.924 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.924 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:21.140089+0000", 2024-04-26T01:09:03.924 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022088199999999999 2024-04-26T01:09:03.924 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.924 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.924 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.924 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.924 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.924 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s4 71/44 ECSubWrite(tid=234, reqid=client.4521.0:741, at_version=71'32, trim_to=0'0, roll_forward_to=71'28))", 2024-04-26T01:09:03.924 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:37.153424+0000", 2024-04-26T01:09:03.925 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 26.706753605999999, 2024-04-26T01:09:03.925 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.162277859, 2024-04-26T01:09:03.925 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.925 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.925 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:09:03.925 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.925 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.925 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.925 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.153424+0000", 2024-04-26T01:09:03.925 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.925 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.925 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.925 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.925 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.153424+0000", 2024-04-26T01:09:03.926 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.926 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.926 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.926 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.926 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.153426+0000", 2024-04-26T01:09:03.926 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8729999999999999e-06 2024-04-26T01:09:03.926 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.926 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.926 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.926 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.153470+0000", 2024-04-26T01:09:03.926 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3185000000000001e-05 2024-04-26T01:09:03.926 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.926 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.926 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.927 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.313873+0000", 2024-04-26T01:09:03.927 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.160403237 2024-04-26T01:09:03.927 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.927 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.927 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.927 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.313888+0000", 2024-04-26T01:09:03.927 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4796e-05 2024-04-26T01:09:03.927 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.927 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.927 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.927 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.314082+0000", 2024-04-26T01:09:03.927 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019490800000000001 2024-04-26T01:09:03.927 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.927 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.928 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.928 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.314120+0000", 2024-04-26T01:09:03.928 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7954999999999998e-05 2024-04-26T01:09:03.928 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.928 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.928 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.928 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.315656+0000", 2024-04-26T01:09:03.928 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0015357960000000001 2024-04-26T01:09:03.928 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.928 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.928 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.928 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.315702+0000", 2024-04-26T01:09:03.928 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6109000000000002e-05 2024-04-26T01:09:03.928 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.929 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.929 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.929 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.929 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.929 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.0s2 71/30 ECSubWrite(tid=78, reqid=client.4521.0:774, at_version=71'64, trim_to=0'0, roll_forward_to=71'61))", 2024-04-26T01:09:03.929 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:37.284626+0000", 2024-04-26T01:09:03.929 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 26.575551894, 2024-04-26T01:09:03.929 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.30429075900000002, 2024-04-26T01:09:03.929 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.929 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.929 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:09:03.929 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.929 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.929 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.930 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.284626+0000", 2024-04-26T01:09:03.930 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.930 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.930 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.930 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.930 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.284626+0000", 2024-04-26T01:09:03.930 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.930 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.930 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.930 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.930 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.284628+0000", 2024-04-26T01:09:03.930 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5570000000000001e-06 2024-04-26T01:09:03.930 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.930 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.931 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.931 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.284659+0000", 2024-04-26T01:09:03.931 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1751999999999997e-05 2024-04-26T01:09:03.931 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.931 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.931 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.931 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.587683+0000", 2024-04-26T01:09:03.931 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.30302400200000001 2024-04-26T01:09:03.931 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.931 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.931 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.931 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.587698+0000", 2024-04-26T01:09:03.931 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5006e-05 2024-04-26T01:09:03.931 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.932 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.932 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.932 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.587817+0000", 2024-04-26T01:09:03.932 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011894499999999999 2024-04-26T01:09:03.932 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.932 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.932 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.932 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.587840+0000", 2024-04-26T01:09:03.932 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2218999999999999e-05 2024-04-26T01:09:03.932 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.932 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.932 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.932 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.588819+0000", 2024-04-26T01:09:03.932 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00097946199999999995 2024-04-26T01:09:03.933 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.933 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.933 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.933 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.588917+0000", 2024-04-26T01:09:03.933 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7816e-05 2024-04-26T01:09:03.933 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.933 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.933 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.933 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.933 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.933 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDECSubOpWrite(3.2s4 71/44 ECSubWrite(tid=238, reqid=client.4521.0:741, at_version=0'0, trim_to=0'0, roll_forward_to=71'32))", 2024-04-26T01:09:03.933 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:37.316333+0000", 2024-04-26T01:09:03.933 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 26.543845107999999, 2024-04-26T01:09:03.933 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.27762489099999998, 2024-04-26T01:09:03.934 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.934 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.934 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2024-04-26T01:09:03.934 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.934 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.934 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.934 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.316333+0000", 2024-04-26T01:09:03.934 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.934 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.934 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.934 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.934 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.316333+0000", 2024-04-26T01:09:03.934 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.934 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.935 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.935 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.935 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.316334+0000", 2024-04-26T01:09:03.935 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4169999999999999e-06 2024-04-26T01:09:03.935 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.935 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.935 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.935 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.316361+0000", 2024-04-26T01:09:03.935 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6299e-05 2024-04-26T01:09:03.935 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.935 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.935 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.935 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.592405+0000", 2024-04-26T01:09:03.935 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.27604479100000001 2024-04-26T01:09:03.935 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.936 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.936 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.936 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.592428+0000", 2024-04-26T01:09:03.936 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2039e-05 2024-04-26T01:09:03.936 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.936 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.936 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.936 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.592591+0000", 2024-04-26T01:09:03.936 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016372700000000001 2024-04-26T01:09:03.936 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.936 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.936 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.936 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.592613+0000", 2024-04-26T01:09:03.936 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2118999999999999e-05 2024-04-26T01:09:03.937 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.937 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.937 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.937 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.593876+0000", 2024-04-26T01:09:03.937 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012630989999999999 2024-04-26T01:09:03.937 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.937 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.937 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.937 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:37.593958+0000", 2024-04-26T01:09:03.937 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.14e-05 2024-04-26T01:09:03.937 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.937 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.937 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.937 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.938 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.938 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:785 3.5s0 3:a8ed6f44:::smithi16428812-28:head [copy-from ver 62] snapc 15=[15] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:09:03.938 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:38.277776+0000", 2024-04-26T01:09:03.938 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 25.582402403, 2024-04-26T01:09:03.938 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.036482200999999999, 2024-04-26T01:09:03.938 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.938 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.938 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:09:03.938 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.938 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.938 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.938 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 785 2024-04-26T01:09:03.938 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.938 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.939 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.939 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.939 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277776+0000", 2024-04-26T01:09:03.939 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.939 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.939 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.939 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.939 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277776+0000", 2024-04-26T01:09:03.939 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.939 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.939 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.939 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.939 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277778+0000", 2024-04-26T01:09:03.939 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.866e-06 2024-04-26T01:09:03.940 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.940 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.940 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.940 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277785+0000", 2024-04-26T01:09:03.940 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2629999999999997e-06 2024-04-26T01:09:03.940 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.940 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.940 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.940 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277800+0000", 2024-04-26T01:09:03.940 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4888999999999999e-05 2024-04-26T01:09:03.940 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.940 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.940 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.940 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277807+0000", 2024-04-26T01:09:03.941 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8170000000000001e-06 2024-04-26T01:09:03.941 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.941 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.941 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.941 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277869+0000", 2024-04-26T01:09:03.941 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2844999999999999e-05 2024-04-26T01:09:03.941 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.941 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.941 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.941 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277990+0000", 2024-04-26T01:09:03.941 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012097899999999999 2024-04-26T01:09:03.941 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.941 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.941 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_started", 2024-04-26T01:09:03.941 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.302496+0000", 2024-04-26T01:09:03.942 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.024505859000000001 2024-04-26T01:09:03.942 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.942 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.942 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_committed", 2024-04-26T01:09:03.942 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.311498+0000", 2024-04-26T01:09:03.942 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0090014369999999993 2024-04-26T01:09:03.942 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.942 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.942 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2024-04-26T01:09:03.942 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.314125+0000", 2024-04-26T01:09:03.942 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0026272930000000002 2024-04-26T01:09:03.942 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.942 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.942 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.943 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.314258+0000", 2024-04-26T01:09:03.943 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000132953 2024-04-26T01:09:03.943 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.943 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.943 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.943 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.943 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.943 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:786 3.5s0 3:a8ed6f44:::smithi16428812-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:09:03.943 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:38.277827+0000", 2024-04-26T01:09:03.943 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 25.582350980000001, 2024-04-26T01:09:03.943 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.033385495000000001, 2024-04-26T01:09:03.943 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.943 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.943 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:09:03.943 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 786 2024-04-26T01:09:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277827+0000", 2024-04-26T01:09:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277827+0000", 2024-04-26T01:09:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277828+0000", 2024-04-26T01:09:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6199999999999997e-07 2024-04-26T01:09:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277832+0000", 2024-04-26T01:09:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1400000000000002e-06 2024-04-26T01:09:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277841+0000", 2024-04-26T01:09:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0699999999999996e-06 2024-04-26T01:09:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.946 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.946 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.946 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.277848+0000", 2024-04-26T01:09:03.946 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1119999999999998e-06 2024-04-26T01:09:03.946 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.946 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.946 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.946 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.278246+0000", 2024-04-26T01:09:03.946 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00039790799999999999 2024-04-26T01:09:03.946 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.946 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.946 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2024-04-26T01:09:03.946 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.278408+0000", 2024-04-26T01:09:03.947 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016165099999999999 2024-04-26T01:09:03.947 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.947 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.947 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.947 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.310714+0000", 2024-04-26T01:09:03.947 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.032306058999999998 2024-04-26T01:09:03.947 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.947 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.947 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.947 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.311023+0000", 2024-04-26T01:09:03.947 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00030885 2024-04-26T01:09:03.947 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.947 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.948 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.948 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:38.311213+0000", 2024-04-26T01:09:03.948 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00018994300000000001 2024-04-26T01:09:03.948 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.948 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.948 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.948 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.948 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.948 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:863 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1753088,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-04-26T01:09:03.948 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:43.278882+0000", 2024-04-26T01:09:03.948 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 20.581295679, 2024-04-26T01:09:03.948 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.018957168, 2024-04-26T01:09:03.948 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.949 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.949 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:09:03.949 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.949 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.949 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.949 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 863 2024-04-26T01:09:03.949 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.949 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.949 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.949 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.949 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278882+0000", 2024-04-26T01:09:03.949 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.949 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.949 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.950 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.950 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278882+0000", 2024-04-26T01:09:03.950 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.950 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.950 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.950 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.950 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278886+0000", 2024-04-26T01:09:03.950 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8829999999999999e-06 2024-04-26T01:09:03.950 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.950 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.950 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.950 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278905+0000", 2024-04-26T01:09:03.950 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9168999999999999e-05 2024-04-26T01:09:03.950 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.950 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.951 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.951 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278950+0000", 2024-04-26T01:09:03.951 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5028000000000003e-05 2024-04-26T01:09:03.951 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.951 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.951 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.951 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.278974+0000", 2024-04-26T01:09:03.951 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3682000000000001e-05 2024-04-26T01:09:03.951 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.951 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.951 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.951 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279142+0000", 2024-04-26T01:09:03.951 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016799499999999999 2024-04-26T01:09:03.951 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.952 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.952 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2024-04-26T01:09:03.952 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279164+0000", 2024-04-26T01:09:03.952 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2039e-05 2024-04-26T01:09:03.952 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.952 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.952 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.952 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.283284+0000", 2024-04-26T01:09:03.952 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0041193239999999997 2024-04-26T01:09:03.952 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.952 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.952 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.952 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.283451+0000", 2024-04-26T01:09:03.952 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016756799999999999 2024-04-26T01:09:03.953 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.953 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.953 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.953 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.297840+0000", 2024-04-26T01:09:03.953 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.01438848 2024-04-26T01:09:03.953 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.953 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.953 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.953 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.953 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.953 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:864 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1753088,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-04-26T01:09:03.953 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:43.279019+0000", 2024-04-26T01:09:03.953 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 20.581158738999999, 2024-04-26T01:09:03.953 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.021489656999999999, 2024-04-26T01:09:03.953 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.954 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.954 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:09:03.954 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.954 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.954 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.954 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 864 2024-04-26T01:09:03.954 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.954 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.954 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.954 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.954 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279019+0000", 2024-04-26T01:09:03.954 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.954 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.954 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.955 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.955 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279019+0000", 2024-04-26T01:09:03.955 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.955 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.955 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.955 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.955 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279021+0000", 2024-04-26T01:09:03.955 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.243e-06 2024-04-26T01:09:03.955 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.955 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.955 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.955 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279028+0000", 2024-04-26T01:09:03.955 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0899999999999999e-06 2024-04-26T01:09:03.955 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.956 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.956 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.956 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279044+0000", 2024-04-26T01:09:03.956 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6545000000000002e-05 2024-04-26T01:09:03.956 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.956 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.956 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.956 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279053+0000", 2024-04-26T01:09:03.956 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1600000000000004e-06 2024-04-26T01:09:03.956 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.956 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.956 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.956 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279342+0000", 2024-04-26T01:09:03.956 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028902300000000001 2024-04-26T01:09:03.956 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.957 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.957 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:09:03.957 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279354+0000", 2024-04-26T01:09:03.957 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1542999999999999e-05 2024-04-26T01:09:03.957 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.957 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.957 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.957 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.284845+0000", 2024-04-26T01:09:03.957 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0054915240000000002 2024-04-26T01:09:03.957 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.957 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.957 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.957 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.284995+0000", 2024-04-26T01:09:03.957 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014962299999999999 2024-04-26T01:09:03.957 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.958 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.958 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.958 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.300509+0000", 2024-04-26T01:09:03.958 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015513905999999999 2024-04-26T01:09:03.958 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.958 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.958 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.958 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.958 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.958 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4521.0:865 3.5s0 3:a661dd24:::smithi16428812-43:head [read 0~1753088,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-04-26T01:09:03.958 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2024-04-26T01:08:43.279067+0000", 2024-04-26T01:09:03.958 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 20.581110981999998, 2024-04-26T01:09:03.958 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.023511127999999999, 2024-04-26T01:09:03.958 INFO:teuthology.orchestra.run.smithi164.stdout: "continuous": false, 2024-04-26T01:09:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2024-04-26T01:09:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2024-04-26T01:09:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2024-04-26T01:09:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4521", 2024-04-26T01:09:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3129949086", 2024-04-26T01:09:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 865 2024-04-26T01:09:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2024-04-26T01:09:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2024-04-26T01:09:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279067+0000", 2024-04-26T01:09:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2024-04-26T01:09:03.960 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279067+0000", 2024-04-26T01:09:03.960 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2024-04-26T01:09:03.960 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.960 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.960 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2024-04-26T01:09:03.960 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279068+0000", 2024-04-26T01:09:03.960 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9699999999999995e-07 2024-04-26T01:09:03.960 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.960 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.960 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2024-04-26T01:09:03.960 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279072+0000", 2024-04-26T01:09:03.960 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2400000000000001e-06 2024-04-26T01:09:03.960 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.960 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.961 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2024-04-26T01:09:03.961 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279082+0000", 2024-04-26T01:09:03.961 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6430000000000007e-06 2024-04-26T01:09:03.961 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.961 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.961 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2024-04-26T01:09:03.961 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279089+0000", 2024-04-26T01:09:03.961 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0690000000000004e-06 2024-04-26T01:09:03.961 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.961 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.961 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.961 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279431+0000", 2024-04-26T01:09:03.961 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034206800000000001 2024-04-26T01:09:03.961 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.962 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.962 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:09:03.962 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.279454+0000", 2024-04-26T01:09:03.962 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3448999999999999e-05 2024-04-26T01:09:03.962 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.962 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.962 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2024-04-26T01:09:03.962 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.287223+0000", 2024-04-26T01:09:03.962 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077688849999999997 2024-04-26T01:09:03.962 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.962 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.962 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2024-04-26T01:09:03.962 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.287470+0000", 2024-04-26T01:09:03.962 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024656199999999997 2024-04-26T01:09:03.963 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-04-26T01:09:03.963 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-04-26T01:09:03.963 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2024-04-26T01:09:03.963 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2024-04-26T01:08:43.302578+0000", 2024-04-26T01:09:03.963 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015108415 2024-04-26T01:09:03.963 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.963 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.963 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.963 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-04-26T01:09:03.963 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-04-26T01:09:03.963 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:09:03.965 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:09:04.090 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:09:04.090 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:09:04.090 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:09:04.091 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:09:04.098 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-04-26T01:09:04.105 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:09:04.221 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:09:04.221 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:09:04.221 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2024-04-26T01:09:04.221 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2024-04-26T01:09:04.221 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:09:04.235 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:09:04.353 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:09:04.354 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2024-04-26T01:09:04.354 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2024-04-26T01:09:04.354 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [] 2024-04-26T01:09:04.354 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:09:04.368 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:09:04.491 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:09:04.491 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:09:04.491 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2024-04-26T01:09:04.491 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:09:04.505 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:09:04.627 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:09:04.627 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2024-04-26T01:09:04.627 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2024-04-26T01:09:04.627 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2024-04-26T01:09:04.627 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:09:04.642 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:09:04.764 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-04-26T01:09:04.764 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2024-04-26T01:09:04.764 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2024-04-26T01:09:04.764 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [] 2024-04-26T01:09:04.764 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-04-26T01:09:07.430 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-26T01:09:07.430 INFO:tasks.thrashosds.thrasher:Adding osd 11 2024-04-26T01:09:07.430 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 11 2024-04-26T01:09:07.755 INFO:tasks.ceph.mon.a.smithi008.stderr:2024-04-26T01:09:07.752+0000 7f99831a5640 -1 mon.a@0(leader).osd e98 definitely_dead 0 2024-04-26T01:09:08.126 INFO:teuthology.orchestra.run.smithi008.stderr:marked in osd.11. 2024-04-26T01:09:08.149 INFO:tasks.thrashosds.thrasher:Added osd 11 2024-04-26T01:09:08.149 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-26T01:09:08.149 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-04-26T01:09:08.149 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 3 2024-04-26T01:09:08.462 INFO:tasks.ceph.mon.a.smithi008.stderr:2024-04-26T01:09:08.460+0000 7f99831a5640 -1 mon.a@0(leader).osd e99 definitely_dead 0 2024-04-26T01:09:09.129 INFO:teuthology.orchestra.run.smithi008.stderr:marked in osd.3. 2024-04-26T01:09:09.156 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-04-26T01:09:09.157 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-26T01:09:09.311 INFO:teuthology.orchestra.run.smithi008.stdout:mon.a: {} 2024-04-26T01:09:09.311 INFO:teuthology.orchestra.run.smithi008.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T01:09:09.329 INFO:teuthology.orchestra.run.smithi008.stdout:mon.b: {} 2024-04-26T01:09:09.329 INFO:teuthology.orchestra.run.smithi008.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T01:09:09.347 INFO:teuthology.orchestra.run.smithi008.stdout:mon.c: {} 2024-04-26T01:09:09.348 INFO:teuthology.orchestra.run.smithi008.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T01:09:09.364 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-26T01:09:10.138 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.0 to 1 (10000) 2024-04-26T01:09:10.165 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-04-26T01:09:11.154 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-26T01:09:11.178 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-26T01:09:12.164 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.4 to 1 (10000) 2024-04-26T01:09:12.191 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-04-26T01:09:13.167 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-26T01:09:13.193 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-04-26T01:09:14.177 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.8 to 1 (10000) 2024-04-26T01:09:14.201 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-04-26T01:09:15.188 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-04-26T01:09:15.214 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-04-26T01:09:16.197 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.12 to 1 (10000) 2024-04-26T01:09:16.227 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-04-26T01:09:17.214 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-04-26T01:09:17.244 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-26T01:09:18.217 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.1 to 1 (10000) 2024-04-26T01:09:18.244 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-26T01:09:19.226 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-26T01:09:19.254 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-26T01:09:20.236 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.5 to 1 (10000) 2024-04-26T01:09:20.260 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-04-26T01:09:21.245 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-26T01:09:21.275 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-04-26T01:09:22.265 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.9 to 1 (10000) 2024-04-26T01:09:22.292 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-04-26T01:09:23.274 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-04-26T01:09:23.301 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-04-26T01:09:24.289 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.13 to 1 (10000) 2024-04-26T01:09:24.314 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-04-26T01:09:25.292 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-04-26T01:09:25.316 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-26T01:09:26.301 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.2 to 1 (10000) 2024-04-26T01:09:26.330 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-04-26T01:09:27.316 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-26T01:09:27.342 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-26T01:09:28.318 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.6 to 1 (10000) 2024-04-26T01:09:28.346 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-04-26T01:09:29.342 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-26T01:09:29.365 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-04-26T01:09:30.350 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.10 to 1 (10000) 2024-04-26T01:09:30.374 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-04-26T01:09:31.353 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-04-26T01:09:31.377 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-04-26T01:09:32.370 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.14 to 1 (10000) 2024-04-26T01:09:32.402 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-04-26T01:09:33.380 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-04-26T01:09:33.405 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-26T01:09:34.395 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.7 to 1 (10000) 2024-04-26T01:09:34.421 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-04-26T01:09:35.398 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-26T01:09:35.421 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-04-26T01:09:36.413 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.15 to 1 (10000) 2024-04-26T01:09:36.442 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-04-26T01:09:37.415 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-04-26T01:09:37.444 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-04-26T01:09:38.430 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.11 to 1 (10000) 2024-04-26T01:09:38.456 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-04-26T01:09:39.439 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-04-26T01:09:39.465 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-26T01:09:40.447 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.3 to 1 (10000) 2024-04-26T01:09:40.470 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-04-26T01:09:41.455 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-26T01:09:41.479 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-26T01:09:41.479 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-26T01:09:41.479 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-26T01:09:41.480 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-26T01:09:41.480 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-26T01:09:41.480 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T01:09:41.791 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:09:41.791 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":132,"fsid":"4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554","created":"2024-04-26T01:07:31.350368+0000","modified":"2024-04-26T01:09:41.447772+0000","last_up_change":"2024-04-26T01:09:03.085260+0000","last_in_change":"2024-04-26T01:09:09.126496+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:07:44.085247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T01:07:46.423491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"abed240a-e896-459b-8a43-cd1419cafbc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6801","nonce":188308042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6803","nonce":188308042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6807","nonce":188308042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6805","nonce":188308042}]},"public_addr":"172.21.15.8:6801/188308042","cluster_addr":"172.21.15.8:6803/188308042","heartbeat_back_addr":"172.21.15.8:6807/188308042","heartbeat_front_addr":"172.21.15.8:6805/188308042","state":["exists","up"]},{"osd":1,"uuid":"831c717c-e124-4489-a44f-93db78c47deb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6819","nonce":3487767810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6823","nonce":3487767810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6831","nonce":3487767810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6828","nonce":3487767810}]},"public_addr":"172.21.15.12:6819/3487767810","cluster_addr":"172.21.15.12:6823/3487767810","heartbeat_back_addr":"172.21.15.12:6831/3487767810","heartbeat_front_addr":"172.21.15.12:6828/3487767810","state":["exists","up"]},{"osd":2,"uuid":"92a95165-ae34-4f29-a7f3-fe97afc26af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6803","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6805","nonce":3599623558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6807","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6809","nonce":3599623558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6815","nonce":3599623558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6813","nonce":3599623558}]},"public_addr":"172.21.15.116:6805/3599623558","cluster_addr":"172.21.15.116:6809/3599623558","heartbeat_back_addr":"172.21.15.116:6815/3599623558","heartbeat_front_addr":"172.21.15.116:6813/3599623558","state":["exists","up"]},{"osd":3,"uuid":"e9d8f764-4951-4de4-92c7-682850852e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":79,"up_from":96,"up_thru":100,"down_at":80,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":115208799},{"type":"v1","addr":"172.21.15.164:6804","nonce":115208799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":115208799},{"type":"v1","addr":"172.21.15.164:6808","nonce":115208799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":115208799},{"type":"v1","addr":"172.21.15.164:6816","nonce":115208799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":115208799},{"type":"v1","addr":"172.21.15.164:6812","nonce":115208799}]},"public_addr":"172.21.15.164:6804/115208799","cluster_addr":"172.21.15.164:6808/115208799","heartbeat_back_addr":"172.21.15.164:6816/115208799","heartbeat_front_addr":"172.21.15.164:6812/115208799","state":["exists","up"]},{"osd":4,"uuid":"7f5e93f8-b1f6-41e7-8da9-b8e57d4259bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6825","nonce":425515720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6827","nonce":425515720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6831","nonce":425515720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6829","nonce":425515720}]},"public_addr":"172.21.15.8:6825/425515720","cluster_addr":"172.21.15.8:6827/425515720","heartbeat_back_addr":"172.21.15.8:6831/425515720","heartbeat_front_addr":"172.21.15.8:6829/425515720","state":["exists","up"]},{"osd":5,"uuid":"f7b3a311-3bf6-4ae5-b7d2-dedbbe6c47ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6809","nonce":1641884791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6811","nonce":1641884791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6815","nonce":1641884791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6813","nonce":1641884791}]},"public_addr":"172.21.15.12:6809/1641884791","cluster_addr":"172.21.15.12:6811/1641884791","heartbeat_back_addr":"172.21.15.12:6815/1641884791","heartbeat_front_addr":"172.21.15.12:6813/1641884791","state":["exists","up"]},{"osd":6,"uuid":"796d7212-13de-43b2-8aa4-4abc4fd210a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6817","nonce":156763037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6820","nonce":156763037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6825","nonce":156763037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6821","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6823","nonce":156763037}]},"public_addr":"172.21.15.116:6817/156763037","cluster_addr":"172.21.15.116:6820/156763037","heartbeat_back_addr":"172.21.15.116:6825/156763037","heartbeat_front_addr":"172.21.15.116:6823/156763037","state":["exists","up"]},{"osd":7,"uuid":"ccb8cd0f-a0d6-44d7-86cc-809eff9687ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6826","nonce":2291946233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6829","nonce":2291946233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6833","nonce":2291946233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6831","nonce":2291946233}]},"public_addr":"172.21.15.164:6826/2291946233","cluster_addr":"172.21.15.164:6829/2291946233","heartbeat_back_addr":"172.21.15.164:6833/2291946233","heartbeat_front_addr":"172.21.15.164:6831/2291946233","state":["exists","up"]},{"osd":8,"uuid":"bf13e311-71b7-44ff-af1b-68d118935c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6809","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6811","nonce":2709504491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6813","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6815","nonce":2709504491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6821","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6823","nonce":2709504491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6817","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6819","nonce":2709504491}]},"public_addr":"172.21.15.8:6811/2709504491","cluster_addr":"172.21.15.8:6815/2709504491","heartbeat_back_addr":"172.21.15.8:6823/2709504491","heartbeat_front_addr":"172.21.15.8:6819/2709504491","state":["exists","up"]},{"osd":9,"uuid":"4864f46f-fd7b-48fc-908b-2564f7bda67b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6801","nonce":1155130892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6803","nonce":1155130892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6807","nonce":1155130892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6805","nonce":1155130892}]},"public_addr":"172.21.15.12:6801/1155130892","cluster_addr":"172.21.15.12:6803/1155130892","heartbeat_back_addr":"172.21.15.12:6807/1155130892","heartbeat_front_addr":"172.21.15.12:6805/1155130892","state":["exists","up"]},{"osd":10,"uuid":"09cbb9ca-f609-409e-821b-93e695b87067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6822","nonce":1461831728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6827","nonce":1461831728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6831","nonce":1461831728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6829","nonce":1461831728}]},"public_addr":"172.21.15.116:6822/1461831728","cluster_addr":"172.21.15.116:6827/1461831728","heartbeat_back_addr":"172.21.15.116:6831/1461831728","heartbeat_front_addr":"172.21.15.116:6829/1461831728","state":["exists","up"]},{"osd":11,"uuid":"9bfdf353-5174-40dd-bb3c-ab2dc850adad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":42,"up_from":60,"up_thru":30,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6819","nonce":838948348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6821","nonce":838948348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6825","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6827","nonce":838948348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6823","nonce":838948348}]},"public_addr":"172.21.15.164:6819/838948348","cluster_addr":"172.21.15.164:6821/838948348","heartbeat_back_addr":"172.21.15.164:6827/838948348","heartbeat_front_addr":"172.21.15.164:6823/838948348","state":["exists","up"]},{"osd":12,"uuid":"a3e244d8-a765-467e-820c-caa5d1d97c9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6810","nonce":2082773890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6814","nonce":2082773890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6822","nonce":2082773890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6818","nonce":2082773890}]},"public_addr":"172.21.15.8:6810/2082773890","cluster_addr":"172.21.15.8:6814/2082773890","heartbeat_back_addr":"172.21.15.8:6822/2082773890","heartbeat_front_addr":"172.21.15.8:6818/2082773890","state":["exists","up"]},{"osd":13,"uuid":"383875c2-0a0c-4834-8beb-a6784eef1f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6817","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6818","nonce":3158583580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6822","nonce":3158583580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6829","nonce":3158583580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6826","nonce":3158583580}]},"public_addr":"172.21.15.12:6818/3158583580","cluster_addr":"172.21.15.12:6822/3158583580","heartbeat_back_addr":"172.21.15.12:6829/3158583580","heartbeat_front_addr":"172.21.15.12:6826/3158583580","state":["exists","up"]},{"osd":14,"uuid":"30311d57-c625-4dff-98d5-1ba0a84c0ceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6801","nonce":2158650188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6804","nonce":2158650188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6812","nonce":2158650188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6808","nonce":2158650188}]},"public_addr":"172.21.15.116:6801/2158650188","cluster_addr":"172.21.15.116:6804/2158650188","heartbeat_back_addr":"172.21.15.116:6812/2158650188","heartbeat_front_addr":"172.21.15.116:6808/2158650188","state":["exists","up"]},{"osd":15,"uuid":"1243478f-9908-460d-88d8-3c542d5258b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6805","nonce":2601606699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6809","nonce":2601606699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6817","nonce":2601606699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6813","nonce":2601606699}]},"public_addr":"172.21.15.164:6805/2601606699","cluster_addr":"172.21.15.164:6809/2601606699","heartbeat_back_addr":"172.21.15.164:6817/2601606699","heartbeat_front_addr":"172.21.15.164:6813/2601606699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:43.889257+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:46.000388+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:07.998091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:43.947103+0000","dead_epoch":80},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:10.476891+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:44.951276+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:47.952421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:11.399059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:51.155841+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:10.299960+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:58.178338+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:01.663819+0000","dead_epoch":43},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:09.972775+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:05.889679+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:47.436227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:05.698647+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":21,"length":1},{"begin":24,"length":2},{"begin":28,"length":3}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:09:41.809 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-26T01:09:41.809 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T01:09:42.116 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:09:42.117 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":132,"fsid":"4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554","created":"2024-04-26T01:07:31.350368+0000","modified":"2024-04-26T01:09:41.447772+0000","last_up_change":"2024-04-26T01:09:03.085260+0000","last_in_change":"2024-04-26T01:09:09.126496+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:07:44.085247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T01:07:46.423491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"abed240a-e896-459b-8a43-cd1419cafbc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6801","nonce":188308042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6803","nonce":188308042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6807","nonce":188308042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6805","nonce":188308042}]},"public_addr":"172.21.15.8:6801/188308042","cluster_addr":"172.21.15.8:6803/188308042","heartbeat_back_addr":"172.21.15.8:6807/188308042","heartbeat_front_addr":"172.21.15.8:6805/188308042","state":["exists","up"]},{"osd":1,"uuid":"831c717c-e124-4489-a44f-93db78c47deb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6819","nonce":3487767810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6823","nonce":3487767810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6831","nonce":3487767810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6828","nonce":3487767810}]},"public_addr":"172.21.15.12:6819/3487767810","cluster_addr":"172.21.15.12:6823/3487767810","heartbeat_back_addr":"172.21.15.12:6831/3487767810","heartbeat_front_addr":"172.21.15.12:6828/3487767810","state":["exists","up"]},{"osd":2,"uuid":"92a95165-ae34-4f29-a7f3-fe97afc26af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6803","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6805","nonce":3599623558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6807","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6809","nonce":3599623558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6815","nonce":3599623558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6813","nonce":3599623558}]},"public_addr":"172.21.15.116:6805/3599623558","cluster_addr":"172.21.15.116:6809/3599623558","heartbeat_back_addr":"172.21.15.116:6815/3599623558","heartbeat_front_addr":"172.21.15.116:6813/3599623558","state":["exists","up"]},{"osd":3,"uuid":"e9d8f764-4951-4de4-92c7-682850852e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":79,"up_from":96,"up_thru":100,"down_at":80,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":115208799},{"type":"v1","addr":"172.21.15.164:6804","nonce":115208799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":115208799},{"type":"v1","addr":"172.21.15.164:6808","nonce":115208799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":115208799},{"type":"v1","addr":"172.21.15.164:6816","nonce":115208799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":115208799},{"type":"v1","addr":"172.21.15.164:6812","nonce":115208799}]},"public_addr":"172.21.15.164:6804/115208799","cluster_addr":"172.21.15.164:6808/115208799","heartbeat_back_addr":"172.21.15.164:6816/115208799","heartbeat_front_addr":"172.21.15.164:6812/115208799","state":["exists","up"]},{"osd":4,"uuid":"7f5e93f8-b1f6-41e7-8da9-b8e57d4259bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6825","nonce":425515720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6827","nonce":425515720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6831","nonce":425515720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6829","nonce":425515720}]},"public_addr":"172.21.15.8:6825/425515720","cluster_addr":"172.21.15.8:6827/425515720","heartbeat_back_addr":"172.21.15.8:6831/425515720","heartbeat_front_addr":"172.21.15.8:6829/425515720","state":["exists","up"]},{"osd":5,"uuid":"f7b3a311-3bf6-4ae5-b7d2-dedbbe6c47ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6809","nonce":1641884791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6811","nonce":1641884791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6815","nonce":1641884791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6813","nonce":1641884791}]},"public_addr":"172.21.15.12:6809/1641884791","cluster_addr":"172.21.15.12:6811/1641884791","heartbeat_back_addr":"172.21.15.12:6815/1641884791","heartbeat_front_addr":"172.21.15.12:6813/1641884791","state":["exists","up"]},{"osd":6,"uuid":"796d7212-13de-43b2-8aa4-4abc4fd210a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6817","nonce":156763037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6820","nonce":156763037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6825","nonce":156763037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6821","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6823","nonce":156763037}]},"public_addr":"172.21.15.116:6817/156763037","cluster_addr":"172.21.15.116:6820/156763037","heartbeat_back_addr":"172.21.15.116:6825/156763037","heartbeat_front_addr":"172.21.15.116:6823/156763037","state":["exists","up"]},{"osd":7,"uuid":"ccb8cd0f-a0d6-44d7-86cc-809eff9687ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6826","nonce":2291946233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6829","nonce":2291946233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6833","nonce":2291946233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6831","nonce":2291946233}]},"public_addr":"172.21.15.164:6826/2291946233","cluster_addr":"172.21.15.164:6829/2291946233","heartbeat_back_addr":"172.21.15.164:6833/2291946233","heartbeat_front_addr":"172.21.15.164:6831/2291946233","state":["exists","up"]},{"osd":8,"uuid":"bf13e311-71b7-44ff-af1b-68d118935c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6809","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6811","nonce":2709504491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6813","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6815","nonce":2709504491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6821","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6823","nonce":2709504491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6817","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6819","nonce":2709504491}]},"public_addr":"172.21.15.8:6811/2709504491","cluster_addr":"172.21.15.8:6815/2709504491","heartbeat_back_addr":"172.21.15.8:6823/2709504491","heartbeat_front_addr":"172.21.15.8:6819/2709504491","state":["exists","up"]},{"osd":9,"uuid":"4864f46f-fd7b-48fc-908b-2564f7bda67b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6801","nonce":1155130892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6803","nonce":1155130892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6807","nonce":1155130892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6805","nonce":1155130892}]},"public_addr":"172.21.15.12:6801/1155130892","cluster_addr":"172.21.15.12:6803/1155130892","heartbeat_back_addr":"172.21.15.12:6807/1155130892","heartbeat_front_addr":"172.21.15.12:6805/1155130892","state":["exists","up"]},{"osd":10,"uuid":"09cbb9ca-f609-409e-821b-93e695b87067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6822","nonce":1461831728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6827","nonce":1461831728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6831","nonce":1461831728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6829","nonce":1461831728}]},"public_addr":"172.21.15.116:6822/1461831728","cluster_addr":"172.21.15.116:6827/1461831728","heartbeat_back_addr":"172.21.15.116:6831/1461831728","heartbeat_front_addr":"172.21.15.116:6829/1461831728","state":["exists","up"]},{"osd":11,"uuid":"9bfdf353-5174-40dd-bb3c-ab2dc850adad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":42,"up_from":60,"up_thru":30,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6819","nonce":838948348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6821","nonce":838948348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6825","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6827","nonce":838948348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6823","nonce":838948348}]},"public_addr":"172.21.15.164:6819/838948348","cluster_addr":"172.21.15.164:6821/838948348","heartbeat_back_addr":"172.21.15.164:6827/838948348","heartbeat_front_addr":"172.21.15.164:6823/838948348","state":["exists","up"]},{"osd":12,"uuid":"a3e244d8-a765-467e-820c-caa5d1d97c9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6810","nonce":2082773890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6814","nonce":2082773890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6822","nonce":2082773890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6818","nonce":2082773890}]},"public_addr":"172.21.15.8:6810/2082773890","cluster_addr":"172.21.15.8:6814/2082773890","heartbeat_back_addr":"172.21.15.8:6822/2082773890","heartbeat_front_addr":"172.21.15.8:6818/2082773890","state":["exists","up"]},{"osd":13,"uuid":"383875c2-0a0c-4834-8beb-a6784eef1f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6817","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6818","nonce":3158583580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6822","nonce":3158583580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6829","nonce":3158583580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6826","nonce":3158583580}]},"public_addr":"172.21.15.12:6818/3158583580","cluster_addr":"172.21.15.12:6822/3158583580","heartbeat_back_addr":"172.21.15.12:6829/3158583580","heartbeat_front_addr":"172.21.15.12:6826/3158583580","state":["exists","up"]},{"osd":14,"uuid":"30311d57-c625-4dff-98d5-1ba0a84c0ceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6801","nonce":2158650188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6804","nonce":2158650188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6812","nonce":2158650188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6808","nonce":2158650188}]},"public_addr":"172.21.15.116:6801/2158650188","cluster_addr":"172.21.15.116:6804/2158650188","heartbeat_back_addr":"172.21.15.116:6812/2158650188","heartbeat_front_addr":"172.21.15.116:6808/2158650188","state":["exists","up"]},{"osd":15,"uuid":"1243478f-9908-460d-88d8-3c542d5258b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6805","nonce":2601606699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6809","nonce":2601606699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6817","nonce":2601606699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6813","nonce":2601606699}]},"public_addr":"172.21.15.164:6805/2601606699","cluster_addr":"172.21.15.164:6809/2601606699","heartbeat_back_addr":"172.21.15.164:6817/2601606699","heartbeat_front_addr":"172.21.15.164:6813/2601606699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:43.889257+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:46.000388+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:07.998091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:43.947103+0000","dead_epoch":80},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:10.476891+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:44.951276+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:47.952421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:11.399059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:51.155841+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:10.299960+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:58.178338+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:01.663819+0000","dead_epoch":43},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:09.972775+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:05.889679+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:47.436227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:05.698647+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":21,"length":1},{"begin":24,"length":2},{"begin":28,"length":3}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:09:42.138 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-26T01:09:42.139 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-26T01:09:42.139 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-26T01:09:42.140 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-26T01:09:42.140 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-26T01:09:42.141 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-26T01:09:42.142 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-26T01:09:42.142 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-26T01:09:42.143 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-04-26T01:09:42.143 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-04-26T01:09:42.143 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-04-26T01:09:42.144 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-04-26T01:09:42.144 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-04-26T01:09:42.145 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-04-26T01:09:42.145 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-04-26T01:09:42.146 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-04-26T01:09:42.539 INFO:teuthology.orchestra.run.smithi008.stdout:90194313242 2024-04-26T01:09:42.540 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T01:09:42.572 INFO:teuthology.orchestra.run.smithi008.stdout:90194313244 2024-04-26T01:09:42.572 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-04-26T01:09:42.596 INFO:teuthology.orchestra.run.smithi008.stdout:90194313242 2024-04-26T01:09:42.596 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-26T01:09:42.619 INFO:teuthology.orchestra.run.smithi008.stdout:90194313242 2024-04-26T01:09:42.619 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-26T01:09:42.627 INFO:teuthology.orchestra.run.smithi008.stdout:90194313243 2024-04-26T01:09:42.627 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T01:09:42.651 INFO:teuthology.orchestra.run.smithi008.stdout:257698037779 2024-04-26T01:09:42.651 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-04-26T01:09:42.656 INFO:teuthology.orchestra.run.smithi008.stdout:94489280538 2024-04-26T01:09:42.657 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-04-26T01:09:42.677 INFO:teuthology.orchestra.run.smithi008.stdout:90194313242 2024-04-26T01:09:42.677 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-04-26T01:09:42.693 INFO:teuthology.orchestra.run.smithi008.stdout:90194313244 2024-04-26T01:09:42.694 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-26T01:09:42.699 INFO:teuthology.orchestra.run.smithi008.stdout:90194313244 2024-04-26T01:09:42.699 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-04-26T01:09:42.716 INFO:teuthology.orchestra.run.smithi008.stdout:90194313243 2024-04-26T01:09:42.716 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-04-26T01:09:42.729 INFO:teuthology.orchestra.run.smithi008.stdout:94489280540 2024-04-26T01:09:42.730 INFO:teuthology.orchestra.run.smithi008.stdout:90194313242 2024-04-26T01:09:42.731 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-04-26T01:09:42.731 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T01:09:42.768 INFO:teuthology.orchestra.run.smithi008.stdout:412316860426 2024-04-26T01:09:42.768 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T01:09:42.770 INFO:teuthology.orchestra.run.smithi008.stdout:94489280540 2024-04-26T01:09:42.770 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-26T01:09:42.791 INFO:teuthology.orchestra.run.smithi008.stdout:90194313243 2024-04-26T01:09:42.792 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-26T01:09:43.217 INFO:teuthology.orchestra.run.smithi008.stderr:2024-04-26T01:09:43.216+0000 7f62311ac640 0 --2- 172.21.15.8:0/2591679507 >> v2:172.21.15.116:3300/0 conn(0x7f622c06e250 0x7f622c0716a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-26T01:09:43.424 INFO:teuthology.orchestra.run.smithi008.stdout:90194313241 2024-04-26T01:09:43.484 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313242 got 90194313241 for osd.0 2024-04-26T01:09:43.529 INFO:teuthology.orchestra.run.smithi008.stdout:257698037778 2024-04-26T01:09:43.584 INFO:tasks.ceph.ceph_manager.ceph:need seq 257698037779 got 257698037778 for osd.11 2024-04-26T01:09:43.608 INFO:teuthology.orchestra.run.smithi008.stdout:90194313241 2024-04-26T01:09:43.620 INFO:teuthology.orchestra.run.smithi008.stdout:90194313241 2024-04-26T01:09:43.624 INFO:teuthology.orchestra.run.smithi008.stdout:94489280537 2024-04-26T01:09:43.632 INFO:teuthology.orchestra.run.smithi008.stdout:90194313241 2024-04-26T01:09:43.646 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313242 got 90194313241 for osd.8 2024-04-26T01:09:43.658 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280538 got 94489280537 for osd.13 2024-04-26T01:09:43.671 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313242 got 90194313241 for osd.6 2024-04-26T01:09:43.683 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313242 got 90194313241 for osd.4 2024-04-26T01:09:43.712 INFO:teuthology.orchestra.run.smithi008.stdout:90194313243 2024-04-26T01:09:43.742 INFO:teuthology.orchestra.run.smithi008.stdout:94489280539 2024-04-26T01:09:43.750 INFO:teuthology.orchestra.run.smithi008.stdout:90194313241 2024-04-26T01:09:43.750 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313244 got 90194313243 for osd.12 2024-04-26T01:09:43.769 INFO:teuthology.orchestra.run.smithi008.stdout:90194313243 2024-04-26T01:09:43.772 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313242 got 90194313241 for osd.10 2024-04-26T01:09:43.774 INFO:teuthology.orchestra.run.smithi008.stdout:90194313242 2024-04-26T01:09:43.787 INFO:teuthology.orchestra.run.smithi008.stdout:90194313243 2024-04-26T01:09:43.791 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280539 for osd.5 2024-04-26T01:09:43.792 INFO:teuthology.orchestra.run.smithi008.stdout:412316860425 2024-04-26T01:09:43.807 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313243 got 90194313242 for osd.2 2024-04-26T01:09:43.817 INFO:teuthology.orchestra.run.smithi008.stdout:90194313242 2024-04-26T01:09:43.818 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313244 got 90194313243 for osd.15 2024-04-26T01:09:43.820 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313244 got 90194313243 for osd.7 2024-04-26T01:09:43.823 INFO:teuthology.orchestra.run.smithi008.stdout:94489280539 2024-04-26T01:09:43.824 INFO:tasks.ceph.ceph_manager.ceph:need seq 412316860426 got 412316860425 for osd.3 2024-04-26T01:09:43.830 INFO:teuthology.orchestra.run.smithi008.stdout:90194313242 2024-04-26T01:09:43.837 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313243 got 90194313242 for osd.9 2024-04-26T01:09:43.846 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280539 for osd.1 2024-04-26T01:09:43.850 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313243 got 90194313242 for osd.14 2024-04-26T01:09:44.486 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T01:09:44.585 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-04-26T01:09:44.646 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-04-26T01:09:44.659 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-04-26T01:09:44.672 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-26T01:09:44.684 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-26T01:09:44.752 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-04-26T01:09:44.773 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-04-26T01:09:44.792 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-26T01:09:44.808 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T01:09:44.819 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-04-26T01:09:44.822 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-26T01:09:44.825 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T01:09:44.838 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-26T01:09:44.846 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T01:09:44.850 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-04-26T01:09:44.918 INFO:teuthology.orchestra.run.smithi008.stdout:90194313242 2024-04-26T01:09:44.978 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313242 got 90194313242 for osd.0 2024-04-26T01:09:44.978 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.377 INFO:teuthology.orchestra.run.smithi008.stdout:90194313242 2024-04-26T01:09:45.380 INFO:teuthology.orchestra.run.smithi008.stdout:257698037779 2024-04-26T01:09:45.386 INFO:teuthology.orchestra.run.smithi008.stdout:94489280538 2024-04-26T01:09:45.436 INFO:tasks.ceph.ceph_manager.ceph:need seq 257698037779 got 257698037779 for osd.11 2024-04-26T01:09:45.436 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.438 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313242 got 90194313242 for osd.8 2024-04-26T01:09:45.438 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.443 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280538 got 94489280538 for osd.13 2024-04-26T01:09:45.443 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.503 INFO:teuthology.orchestra.run.smithi008.stdout:90194313242 2024-04-26T01:09:45.524 INFO:teuthology.orchestra.run.smithi008.stdout:90194313242 2024-04-26T01:09:45.551 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313242 got 90194313242 for osd.4 2024-04-26T01:09:45.551 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.562 INFO:teuthology.orchestra.run.smithi008.stdout:90194313243 2024-04-26T01:09:45.601 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313242 got 90194313242 for osd.6 2024-04-26T01:09:45.601 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.616 INFO:teuthology.orchestra.run.smithi008.stdout:90194313244 2024-04-26T01:09:45.616 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313242 got 90194313243 for osd.10 2024-04-26T01:09:45.617 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.657 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313244 got 90194313244 for osd.12 2024-04-26T01:09:45.657 INFO:teuthology.orchestra.run.smithi008.stdout:94489280540 2024-04-26T01:09:45.657 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.687 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280540 for osd.5 2024-04-26T01:09:45.688 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.709 INFO:teuthology.orchestra.run.smithi008.stdout:90194313243 2024-04-26T01:09:45.726 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313243 got 90194313243 for osd.14 2024-04-26T01:09:45.727 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.727 INFO:teuthology.orchestra.run.smithi008.stdout:90194313244 2024-04-26T01:09:45.736 INFO:teuthology.orchestra.run.smithi008.stdout:90194313243 2024-04-26T01:09:45.737 INFO:teuthology.orchestra.run.smithi008.stdout:412316860427 2024-04-26T01:09:45.749 INFO:teuthology.orchestra.run.smithi008.stdout:90194313243 2024-04-26T01:09:45.753 INFO:teuthology.orchestra.run.smithi008.stdout:90194313244 2024-04-26T01:09:45.754 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313244 got 90194313244 for osd.7 2024-04-26T01:09:45.754 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.763 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313243 got 90194313243 for osd.9 2024-04-26T01:09:45.763 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.770 INFO:tasks.ceph.ceph_manager.ceph:need seq 412316860426 got 412316860427 for osd.3 2024-04-26T01:09:45.770 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.777 INFO:teuthology.orchestra.run.smithi008.stdout:94489280540 2024-04-26T01:09:45.779 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313243 got 90194313243 for osd.2 2024-04-26T01:09:45.779 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.781 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313244 got 90194313244 for osd.15 2024-04-26T01:09:45.781 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.794 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280540 for osd.1 2024-04-26T01:09:45.794 DEBUG:teuthology.parallel:result is None 2024-04-26T01:09:45.794 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-26T01:09:45.795 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:09:46.124 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:09:46.124 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-26T01:09:46.140 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-04-26T01:09:46.018011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":538512,"kb_used_data":24428,"kb_used_omap":25,"kb_used_meta":513830,"kb_avail":1509410928,"statfs":{"total":1546188226560,"available":1545636790272,"internally_reserved":0,"allocated":25014272,"data_stored":19815646,"data_compressed":5371375,"data_compressed_allocated":7286784,"data_compressed_original":15433728,"omap_allocated":26158,"internal_metadata":526162386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.576032"},"pg_stats":[{"pgid":"1.0","version":"27'157","reported_seq":426,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.461652+0000","last_change":"2024-04-26T01:09:12.876970+0000","last_active":"2024-04-26T01:09:41.461652+0000","last_peered":"2024-04-26T01:09:41.461652+0000","last_clean":"2024-04-26T01:09:41.461652+0000","last_became_active":"2024-04-26T01:07:46.407301+0000","last_became_peered":"2024-04-26T01:07:46.407301+0000","last_unstale":"2024-04-26T01:09:41.461652+0000","last_undegraded":"2024-04-26T01:09:41.461652+0000","last_fullsized":"2024-04-26T01:09:41.461652+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'157","last_scrub_stamp":"2024-04-26T01:09:12.876525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_clean_scrub_stamp":"2024-04-26T01:09:12.876525+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:35.766266+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":225,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.461874+0000","last_change":"2024-04-26T01:09:08.691918+0000","last_active":"2024-04-26T01:09:41.461874+0000","last_peered":"2024-04-26T01:09:41.461874+0000","last_clean":"2024-04-26T01:09:41.461874+0000","last_became_active":"2024-04-26T01:07:48.144313+0000","last_became_peered":"2024-04-26T01:07:48.144313+0000","last_unstale":"2024-04-26T01:09:41.461874+0000","last_undegraded":"2024-04-26T01:09:41.461874+0000","last_fullsized":"2024-04-26T01:09:41.461874+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:08.691698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:08.691698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:29.117900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":71,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.726383+0000","last_change":"2024-04-26T01:09:10.395016+0000","last_active":"2024-04-26T01:09:41.726383+0000","last_peered":"2024-04-26T01:09:41.726383+0000","last_clean":"2024-04-26T01:09:41.726383+0000","last_became_active":"2024-04-26T01:09:10.394370+0000","last_became_peered":"2024-04-26T01:09:10.394370+0000","last_unstale":"2024-04-26T01:09:41.726383+0000","last_undegraded":"2024-04-26T01:09:41.726383+0000","last_fullsized":"2024-04-26T01:09:41.726383+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:08.921880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:08.921880+0000","last_clean_scrub_stamp":"2024-04-26T01:09:08.921880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:18.094815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":226,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.461666+0000","last_change":"2024-04-26T01:09:15.910993+0000","last_active":"2024-04-26T01:09:41.461666+0000","last_peered":"2024-04-26T01:09:41.461666+0000","last_clean":"2024-04-26T01:09:41.461666+0000","last_became_active":"2024-04-26T01:07:48.145991+0000","last_became_peered":"2024-04-26T01:07:48.145991+0000","last_unstale":"2024-04-26T01:09:41.461666+0000","last_undegraded":"2024-04-26T01:09:41.461666+0000","last_fullsized":"2024-04-26T01:09:41.461666+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:15.910744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:15.910744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:35.650953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":79,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.726383+0000","last_change":"2024-04-26T01:09:12.345740+0000","last_active":"2024-04-26T01:09:41.726383+0000","last_peered":"2024-04-26T01:09:41.726383+0000","last_clean":"2024-04-26T01:09:41.726383+0000","last_became_active":"2024-04-26T01:09:10.896103+0000","last_became_peered":"2024-04-26T01:09:10.896103+0000","last_unstale":"2024-04-26T01:09:41.726383+0000","last_undegraded":"2024-04-26T01:09:41.726383+0000","last_fullsized":"2024-04-26T01:09:41.726383+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:12.345508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:12.345508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:13.867577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":228,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.459378+0000","last_change":"2024-04-26T01:08:50.478667+0000","last_active":"2024-04-26T01:09:41.459378+0000","last_peered":"2024-04-26T01:09:41.459378+0000","last_clean":"2024-04-26T01:09:41.459378+0000","last_became_active":"2024-04-26T01:07:48.465232+0000","last_became_peered":"2024-04-26T01:07:48.465232+0000","last_unstale":"2024-04-26T01:09:41.459378+0000","last_undegraded":"2024-04-26T01:09:41.459378+0000","last_fullsized":"2024-04-26T01:09:41.459378+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:08:50.478571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:08:50.478571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:08.707444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":228,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.501932+0000","last_change":"2024-04-26T01:09:14.870357+0000","last_active":"2024-04-26T01:09:41.501932+0000","last_peered":"2024-04-26T01:09:41.501932+0000","last_clean":"2024-04-26T01:09:41.501932+0000","last_became_active":"2024-04-26T01:07:48.145376+0000","last_became_peered":"2024-04-26T01:07:48.145376+0000","last_unstale":"2024-04-26T01:09:41.501932+0000","last_undegraded":"2024-04-26T01:09:41.501932+0000","last_fullsized":"2024-04-26T01:09:41.501932+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:14.870031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:14.870031+0000","last_clean_scrub_stamp":"2024-04-26T01:09:14.870031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:26.187184+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":227,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:42.929264+0000","last_change":"2024-04-26T01:09:17.636007+0000","last_active":"2024-04-26T01:09:42.929264+0000","last_peered":"2024-04-26T01:09:42.929264+0000","last_clean":"2024-04-26T01:09:42.929264+0000","last_became_active":"2024-04-26T01:07:48.145512+0000","last_became_peered":"2024-04-26T01:07:48.145512+0000","last_unstale":"2024-04-26T01:09:42.929264+0000","last_undegraded":"2024-04-26T01:09:42.929264+0000","last_fullsized":"2024-04-26T01:09:42.929264+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:17.635797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:17.635797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:34.968710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":226,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:42.309414+0000","last_change":"2024-04-26T01:09:07.988749+0000","last_active":"2024-04-26T01:09:42.309414+0000","last_peered":"2024-04-26T01:09:42.309414+0000","last_clean":"2024-04-26T01:09:42.309414+0000","last_became_active":"2024-04-26T01:07:48.148023+0000","last_became_peered":"2024-04-26T01:07:48.148023+0000","last_unstale":"2024-04-26T01:09:42.309414+0000","last_undegraded":"2024-04-26T01:09:42.309414+0000","last_fullsized":"2024-04-26T01:09:42.309414+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:07.988439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:07.988439+0000","last_clean_scrub_stamp":"2024-04-26T01:09:07.988439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:14.276176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":23216,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,4,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1617920,"data_stored":1701891,"data_compressed":346059,"data_compressed_allocated":491520,"data_compressed_original":1413120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,8,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54908,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316932,"statfs":{"total":96636764160,"available":96580538368,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313244,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32760,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1617920,"data_stored":1701891,"data_compressed":346059,"data_compressed_allocated":491520,"data_compressed_original":1413120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313243,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313243,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":21,"seq":90194313243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313244,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,3,4,5,6,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313243,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28540,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343300,"statfs":{"total":96636764160,"available":96607539200,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":60,"seq":257698037779,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29232,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27710,"kb_avail":94342608,"statfs":{"total":96636764160,"available":96606830592,"internally_reserved":0,"allocated":1544192,"data_stored":1163345,"data_compressed":331394,"data_compressed_allocated":446464,"data_compressed_original":892928,"omap_allocated":1802,"internal_metadata":28375286},"hb_peers":[0,2,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313245,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[5,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313244,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":96,"seq":412316860427,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29120,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27582,"kb_avail":94342720,"statfs":{"total":96636764160,"available":96606945280,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":2023,"internal_metadata":28243993},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:09:46.141 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:09:46.430 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:09:46.431 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-26T01:09:46.447 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-04-26T01:09:46.018011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":538512,"kb_used_data":24428,"kb_used_omap":25,"kb_used_meta":513830,"kb_avail":1509410928,"statfs":{"total":1546188226560,"available":1545636790272,"internally_reserved":0,"allocated":25014272,"data_stored":19815646,"data_compressed":5371375,"data_compressed_allocated":7286784,"data_compressed_original":15433728,"omap_allocated":26158,"internal_metadata":526162386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.576032"},"pg_stats":[{"pgid":"1.0","version":"27'157","reported_seq":426,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.461652+0000","last_change":"2024-04-26T01:09:12.876970+0000","last_active":"2024-04-26T01:09:41.461652+0000","last_peered":"2024-04-26T01:09:41.461652+0000","last_clean":"2024-04-26T01:09:41.461652+0000","last_became_active":"2024-04-26T01:07:46.407301+0000","last_became_peered":"2024-04-26T01:07:46.407301+0000","last_unstale":"2024-04-26T01:09:41.461652+0000","last_undegraded":"2024-04-26T01:09:41.461652+0000","last_fullsized":"2024-04-26T01:09:41.461652+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'157","last_scrub_stamp":"2024-04-26T01:09:12.876525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_clean_scrub_stamp":"2024-04-26T01:09:12.876525+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:35.766266+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":225,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.461874+0000","last_change":"2024-04-26T01:09:08.691918+0000","last_active":"2024-04-26T01:09:41.461874+0000","last_peered":"2024-04-26T01:09:41.461874+0000","last_clean":"2024-04-26T01:09:41.461874+0000","last_became_active":"2024-04-26T01:07:48.144313+0000","last_became_peered":"2024-04-26T01:07:48.144313+0000","last_unstale":"2024-04-26T01:09:41.461874+0000","last_undegraded":"2024-04-26T01:09:41.461874+0000","last_fullsized":"2024-04-26T01:09:41.461874+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:08.691698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:08.691698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:29.117900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":71,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.726383+0000","last_change":"2024-04-26T01:09:10.395016+0000","last_active":"2024-04-26T01:09:41.726383+0000","last_peered":"2024-04-26T01:09:41.726383+0000","last_clean":"2024-04-26T01:09:41.726383+0000","last_became_active":"2024-04-26T01:09:10.394370+0000","last_became_peered":"2024-04-26T01:09:10.394370+0000","last_unstale":"2024-04-26T01:09:41.726383+0000","last_undegraded":"2024-04-26T01:09:41.726383+0000","last_fullsized":"2024-04-26T01:09:41.726383+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:08.921880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:08.921880+0000","last_clean_scrub_stamp":"2024-04-26T01:09:08.921880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:18.094815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":226,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.461666+0000","last_change":"2024-04-26T01:09:15.910993+0000","last_active":"2024-04-26T01:09:41.461666+0000","last_peered":"2024-04-26T01:09:41.461666+0000","last_clean":"2024-04-26T01:09:41.461666+0000","last_became_active":"2024-04-26T01:07:48.145991+0000","last_became_peered":"2024-04-26T01:07:48.145991+0000","last_unstale":"2024-04-26T01:09:41.461666+0000","last_undegraded":"2024-04-26T01:09:41.461666+0000","last_fullsized":"2024-04-26T01:09:41.461666+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:15.910744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:15.910744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:35.650953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":79,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.726383+0000","last_change":"2024-04-26T01:09:12.345740+0000","last_active":"2024-04-26T01:09:41.726383+0000","last_peered":"2024-04-26T01:09:41.726383+0000","last_clean":"2024-04-26T01:09:41.726383+0000","last_became_active":"2024-04-26T01:09:10.896103+0000","last_became_peered":"2024-04-26T01:09:10.896103+0000","last_unstale":"2024-04-26T01:09:41.726383+0000","last_undegraded":"2024-04-26T01:09:41.726383+0000","last_fullsized":"2024-04-26T01:09:41.726383+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:12.345508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:12.345508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:13.867577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":228,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.459378+0000","last_change":"2024-04-26T01:08:50.478667+0000","last_active":"2024-04-26T01:09:41.459378+0000","last_peered":"2024-04-26T01:09:41.459378+0000","last_clean":"2024-04-26T01:09:41.459378+0000","last_became_active":"2024-04-26T01:07:48.465232+0000","last_became_peered":"2024-04-26T01:07:48.465232+0000","last_unstale":"2024-04-26T01:09:41.459378+0000","last_undegraded":"2024-04-26T01:09:41.459378+0000","last_fullsized":"2024-04-26T01:09:41.459378+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:08:50.478571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:08:50.478571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:08.707444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":228,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.501932+0000","last_change":"2024-04-26T01:09:14.870357+0000","last_active":"2024-04-26T01:09:41.501932+0000","last_peered":"2024-04-26T01:09:41.501932+0000","last_clean":"2024-04-26T01:09:41.501932+0000","last_became_active":"2024-04-26T01:07:48.145376+0000","last_became_peered":"2024-04-26T01:07:48.145376+0000","last_unstale":"2024-04-26T01:09:41.501932+0000","last_undegraded":"2024-04-26T01:09:41.501932+0000","last_fullsized":"2024-04-26T01:09:41.501932+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:14.870031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:14.870031+0000","last_clean_scrub_stamp":"2024-04-26T01:09:14.870031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:26.187184+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":227,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:42.929264+0000","last_change":"2024-04-26T01:09:17.636007+0000","last_active":"2024-04-26T01:09:42.929264+0000","last_peered":"2024-04-26T01:09:42.929264+0000","last_clean":"2024-04-26T01:09:42.929264+0000","last_became_active":"2024-04-26T01:07:48.145512+0000","last_became_peered":"2024-04-26T01:07:48.145512+0000","last_unstale":"2024-04-26T01:09:42.929264+0000","last_undegraded":"2024-04-26T01:09:42.929264+0000","last_fullsized":"2024-04-26T01:09:42.929264+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:17.635797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:17.635797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:34.968710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":226,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:42.309414+0000","last_change":"2024-04-26T01:09:07.988749+0000","last_active":"2024-04-26T01:09:42.309414+0000","last_peered":"2024-04-26T01:09:42.309414+0000","last_clean":"2024-04-26T01:09:42.309414+0000","last_became_active":"2024-04-26T01:07:48.148023+0000","last_became_peered":"2024-04-26T01:07:48.148023+0000","last_unstale":"2024-04-26T01:09:42.309414+0000","last_undegraded":"2024-04-26T01:09:42.309414+0000","last_fullsized":"2024-04-26T01:09:42.309414+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:07.988439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:07.988439+0000","last_clean_scrub_stamp":"2024-04-26T01:09:07.988439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:14.276176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":23216,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,4,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1617920,"data_stored":1701891,"data_compressed":346059,"data_compressed_allocated":491520,"data_compressed_original":1413120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,8,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54908,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316932,"statfs":{"total":96636764160,"available":96580538368,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313244,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32760,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1617920,"data_stored":1701891,"data_compressed":346059,"data_compressed_allocated":491520,"data_compressed_original":1413120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313243,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313243,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":21,"seq":90194313243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313244,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,3,4,5,6,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313243,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28540,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343300,"statfs":{"total":96636764160,"available":96607539200,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":60,"seq":257698037779,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29232,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27710,"kb_avail":94342608,"statfs":{"total":96636764160,"available":96606830592,"internally_reserved":0,"allocated":1544192,"data_stored":1163345,"data_compressed":331394,"data_compressed_allocated":446464,"data_compressed_original":892928,"omap_allocated":1802,"internal_metadata":28375286},"hb_peers":[0,2,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313245,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[5,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313244,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":96,"seq":412316860427,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29120,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27582,"kb_avail":94342720,"statfs":{"total":96636764160,"available":96606945280,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":2023,"internal_metadata":28243993},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:09:46.448 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-26T01:09:46.448 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-26T01:09:46.760 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-26T01:09:46.770 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T01:09:46.770 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:09:47.069 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:09:47.070 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-26T01:09:47.086 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-04-26T01:09:46.018011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":538512,"kb_used_data":24428,"kb_used_omap":25,"kb_used_meta":513830,"kb_avail":1509410928,"statfs":{"total":1546188226560,"available":1545636790272,"internally_reserved":0,"allocated":25014272,"data_stored":19815646,"data_compressed":5371375,"data_compressed_allocated":7286784,"data_compressed_original":15433728,"omap_allocated":26158,"internal_metadata":526162386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.576032"},"pg_stats":[{"pgid":"1.0","version":"27'157","reported_seq":426,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.461652+0000","last_change":"2024-04-26T01:09:12.876970+0000","last_active":"2024-04-26T01:09:41.461652+0000","last_peered":"2024-04-26T01:09:41.461652+0000","last_clean":"2024-04-26T01:09:41.461652+0000","last_became_active":"2024-04-26T01:07:46.407301+0000","last_became_peered":"2024-04-26T01:07:46.407301+0000","last_unstale":"2024-04-26T01:09:41.461652+0000","last_undegraded":"2024-04-26T01:09:41.461652+0000","last_fullsized":"2024-04-26T01:09:41.461652+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'157","last_scrub_stamp":"2024-04-26T01:09:12.876525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_clean_scrub_stamp":"2024-04-26T01:09:12.876525+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:35.766266+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":225,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.461874+0000","last_change":"2024-04-26T01:09:08.691918+0000","last_active":"2024-04-26T01:09:41.461874+0000","last_peered":"2024-04-26T01:09:41.461874+0000","last_clean":"2024-04-26T01:09:41.461874+0000","last_became_active":"2024-04-26T01:07:48.144313+0000","last_became_peered":"2024-04-26T01:07:48.144313+0000","last_unstale":"2024-04-26T01:09:41.461874+0000","last_undegraded":"2024-04-26T01:09:41.461874+0000","last_fullsized":"2024-04-26T01:09:41.461874+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:08.691698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:08.691698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:29.117900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":71,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.726383+0000","last_change":"2024-04-26T01:09:10.395016+0000","last_active":"2024-04-26T01:09:41.726383+0000","last_peered":"2024-04-26T01:09:41.726383+0000","last_clean":"2024-04-26T01:09:41.726383+0000","last_became_active":"2024-04-26T01:09:10.394370+0000","last_became_peered":"2024-04-26T01:09:10.394370+0000","last_unstale":"2024-04-26T01:09:41.726383+0000","last_undegraded":"2024-04-26T01:09:41.726383+0000","last_fullsized":"2024-04-26T01:09:41.726383+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:08.921880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:08.921880+0000","last_clean_scrub_stamp":"2024-04-26T01:09:08.921880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:18.094815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":226,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.461666+0000","last_change":"2024-04-26T01:09:15.910993+0000","last_active":"2024-04-26T01:09:41.461666+0000","last_peered":"2024-04-26T01:09:41.461666+0000","last_clean":"2024-04-26T01:09:41.461666+0000","last_became_active":"2024-04-26T01:07:48.145991+0000","last_became_peered":"2024-04-26T01:07:48.145991+0000","last_unstale":"2024-04-26T01:09:41.461666+0000","last_undegraded":"2024-04-26T01:09:41.461666+0000","last_fullsized":"2024-04-26T01:09:41.461666+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:15.910744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:15.910744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:35.650953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":79,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.726383+0000","last_change":"2024-04-26T01:09:12.345740+0000","last_active":"2024-04-26T01:09:41.726383+0000","last_peered":"2024-04-26T01:09:41.726383+0000","last_clean":"2024-04-26T01:09:41.726383+0000","last_became_active":"2024-04-26T01:09:10.896103+0000","last_became_peered":"2024-04-26T01:09:10.896103+0000","last_unstale":"2024-04-26T01:09:41.726383+0000","last_undegraded":"2024-04-26T01:09:41.726383+0000","last_fullsized":"2024-04-26T01:09:41.726383+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:12.345508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:12.345508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:13.867577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":228,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.459378+0000","last_change":"2024-04-26T01:08:50.478667+0000","last_active":"2024-04-26T01:09:41.459378+0000","last_peered":"2024-04-26T01:09:41.459378+0000","last_clean":"2024-04-26T01:09:41.459378+0000","last_became_active":"2024-04-26T01:07:48.465232+0000","last_became_peered":"2024-04-26T01:07:48.465232+0000","last_unstale":"2024-04-26T01:09:41.459378+0000","last_undegraded":"2024-04-26T01:09:41.459378+0000","last_fullsized":"2024-04-26T01:09:41.459378+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:08:50.478571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:08:50.478571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:08.707444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":228,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.501932+0000","last_change":"2024-04-26T01:09:14.870357+0000","last_active":"2024-04-26T01:09:41.501932+0000","last_peered":"2024-04-26T01:09:41.501932+0000","last_clean":"2024-04-26T01:09:41.501932+0000","last_became_active":"2024-04-26T01:07:48.145376+0000","last_became_peered":"2024-04-26T01:07:48.145376+0000","last_unstale":"2024-04-26T01:09:41.501932+0000","last_undegraded":"2024-04-26T01:09:41.501932+0000","last_fullsized":"2024-04-26T01:09:41.501932+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:14.870031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:14.870031+0000","last_clean_scrub_stamp":"2024-04-26T01:09:14.870031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:26.187184+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":227,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:42.929264+0000","last_change":"2024-04-26T01:09:17.636007+0000","last_active":"2024-04-26T01:09:42.929264+0000","last_peered":"2024-04-26T01:09:42.929264+0000","last_clean":"2024-04-26T01:09:42.929264+0000","last_became_active":"2024-04-26T01:07:48.145512+0000","last_became_peered":"2024-04-26T01:07:48.145512+0000","last_unstale":"2024-04-26T01:09:42.929264+0000","last_undegraded":"2024-04-26T01:09:42.929264+0000","last_fullsized":"2024-04-26T01:09:42.929264+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:17.635797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:17.635797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:34.968710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":226,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:42.309414+0000","last_change":"2024-04-26T01:09:07.988749+0000","last_active":"2024-04-26T01:09:42.309414+0000","last_peered":"2024-04-26T01:09:42.309414+0000","last_clean":"2024-04-26T01:09:42.309414+0000","last_became_active":"2024-04-26T01:07:48.148023+0000","last_became_peered":"2024-04-26T01:07:48.148023+0000","last_unstale":"2024-04-26T01:09:42.309414+0000","last_undegraded":"2024-04-26T01:09:42.309414+0000","last_fullsized":"2024-04-26T01:09:42.309414+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:07.988439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:07.988439+0000","last_clean_scrub_stamp":"2024-04-26T01:09:07.988439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:14.276176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":23216,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,4,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1617920,"data_stored":1701891,"data_compressed":346059,"data_compressed_allocated":491520,"data_compressed_original":1413120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,8,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54908,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316932,"statfs":{"total":96636764160,"available":96580538368,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313244,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32760,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1617920,"data_stored":1701891,"data_compressed":346059,"data_compressed_allocated":491520,"data_compressed_original":1413120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313243,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313243,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":21,"seq":90194313243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313244,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,3,4,5,6,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313243,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28540,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343300,"statfs":{"total":96636764160,"available":96607539200,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":60,"seq":257698037779,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29232,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27710,"kb_avail":94342608,"statfs":{"total":96636764160,"available":96606830592,"internally_reserved":0,"allocated":1544192,"data_stored":1163345,"data_compressed":331394,"data_compressed_allocated":446464,"data_compressed_original":892928,"omap_allocated":1802,"internal_metadata":28375286},"hb_peers":[0,2,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313245,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[5,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313244,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":96,"seq":412316860427,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29120,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27582,"kb_avail":94342720,"statfs":{"total":96636764160,"available":96606945280,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":2023,"internal_metadata":28243993},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:09:47.087 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:09:47.384 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:09:47.384 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-26T01:09:47.402 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-04-26T01:09:46.018011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":538512,"kb_used_data":24428,"kb_used_omap":25,"kb_used_meta":513830,"kb_avail":1509410928,"statfs":{"total":1546188226560,"available":1545636790272,"internally_reserved":0,"allocated":25014272,"data_stored":19815646,"data_compressed":5371375,"data_compressed_allocated":7286784,"data_compressed_original":15433728,"omap_allocated":26158,"internal_metadata":526162386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.576032"},"pg_stats":[{"pgid":"1.0","version":"27'157","reported_seq":426,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.461652+0000","last_change":"2024-04-26T01:09:12.876970+0000","last_active":"2024-04-26T01:09:41.461652+0000","last_peered":"2024-04-26T01:09:41.461652+0000","last_clean":"2024-04-26T01:09:41.461652+0000","last_became_active":"2024-04-26T01:07:46.407301+0000","last_became_peered":"2024-04-26T01:07:46.407301+0000","last_unstale":"2024-04-26T01:09:41.461652+0000","last_undegraded":"2024-04-26T01:09:41.461652+0000","last_fullsized":"2024-04-26T01:09:41.461652+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'157","last_scrub_stamp":"2024-04-26T01:09:12.876525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_clean_scrub_stamp":"2024-04-26T01:09:12.876525+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:35.766266+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":225,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.461874+0000","last_change":"2024-04-26T01:09:08.691918+0000","last_active":"2024-04-26T01:09:41.461874+0000","last_peered":"2024-04-26T01:09:41.461874+0000","last_clean":"2024-04-26T01:09:41.461874+0000","last_became_active":"2024-04-26T01:07:48.144313+0000","last_became_peered":"2024-04-26T01:07:48.144313+0000","last_unstale":"2024-04-26T01:09:41.461874+0000","last_undegraded":"2024-04-26T01:09:41.461874+0000","last_fullsized":"2024-04-26T01:09:41.461874+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:08.691698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:08.691698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:29.117900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":71,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.726383+0000","last_change":"2024-04-26T01:09:10.395016+0000","last_active":"2024-04-26T01:09:41.726383+0000","last_peered":"2024-04-26T01:09:41.726383+0000","last_clean":"2024-04-26T01:09:41.726383+0000","last_became_active":"2024-04-26T01:09:10.394370+0000","last_became_peered":"2024-04-26T01:09:10.394370+0000","last_unstale":"2024-04-26T01:09:41.726383+0000","last_undegraded":"2024-04-26T01:09:41.726383+0000","last_fullsized":"2024-04-26T01:09:41.726383+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:08.921880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:08.921880+0000","last_clean_scrub_stamp":"2024-04-26T01:09:08.921880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:18.094815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":226,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.461666+0000","last_change":"2024-04-26T01:09:15.910993+0000","last_active":"2024-04-26T01:09:41.461666+0000","last_peered":"2024-04-26T01:09:41.461666+0000","last_clean":"2024-04-26T01:09:41.461666+0000","last_became_active":"2024-04-26T01:07:48.145991+0000","last_became_peered":"2024-04-26T01:07:48.145991+0000","last_unstale":"2024-04-26T01:09:41.461666+0000","last_undegraded":"2024-04-26T01:09:41.461666+0000","last_fullsized":"2024-04-26T01:09:41.461666+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:15.910744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:15.910744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:35.650953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":79,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.726383+0000","last_change":"2024-04-26T01:09:12.345740+0000","last_active":"2024-04-26T01:09:41.726383+0000","last_peered":"2024-04-26T01:09:41.726383+0000","last_clean":"2024-04-26T01:09:41.726383+0000","last_became_active":"2024-04-26T01:09:10.896103+0000","last_became_peered":"2024-04-26T01:09:10.896103+0000","last_unstale":"2024-04-26T01:09:41.726383+0000","last_undegraded":"2024-04-26T01:09:41.726383+0000","last_fullsized":"2024-04-26T01:09:41.726383+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:12.345508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:12.345508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:13.867577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":228,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.459378+0000","last_change":"2024-04-26T01:08:50.478667+0000","last_active":"2024-04-26T01:09:41.459378+0000","last_peered":"2024-04-26T01:09:41.459378+0000","last_clean":"2024-04-26T01:09:41.459378+0000","last_became_active":"2024-04-26T01:07:48.465232+0000","last_became_peered":"2024-04-26T01:07:48.465232+0000","last_unstale":"2024-04-26T01:09:41.459378+0000","last_undegraded":"2024-04-26T01:09:41.459378+0000","last_fullsized":"2024-04-26T01:09:41.459378+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:08:50.478571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:08:50.478571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:08.707444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":228,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.501932+0000","last_change":"2024-04-26T01:09:14.870357+0000","last_active":"2024-04-26T01:09:41.501932+0000","last_peered":"2024-04-26T01:09:41.501932+0000","last_clean":"2024-04-26T01:09:41.501932+0000","last_became_active":"2024-04-26T01:07:48.145376+0000","last_became_peered":"2024-04-26T01:07:48.145376+0000","last_unstale":"2024-04-26T01:09:41.501932+0000","last_undegraded":"2024-04-26T01:09:41.501932+0000","last_fullsized":"2024-04-26T01:09:41.501932+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:14.870031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:14.870031+0000","last_clean_scrub_stamp":"2024-04-26T01:09:14.870031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:26.187184+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":227,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:42.929264+0000","last_change":"2024-04-26T01:09:17.636007+0000","last_active":"2024-04-26T01:09:42.929264+0000","last_peered":"2024-04-26T01:09:42.929264+0000","last_clean":"2024-04-26T01:09:42.929264+0000","last_became_active":"2024-04-26T01:07:48.145512+0000","last_became_peered":"2024-04-26T01:07:48.145512+0000","last_unstale":"2024-04-26T01:09:42.929264+0000","last_undegraded":"2024-04-26T01:09:42.929264+0000","last_fullsized":"2024-04-26T01:09:42.929264+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:17.635797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:17.635797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:34.968710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":226,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:42.309414+0000","last_change":"2024-04-26T01:09:07.988749+0000","last_active":"2024-04-26T01:09:42.309414+0000","last_peered":"2024-04-26T01:09:42.309414+0000","last_clean":"2024-04-26T01:09:42.309414+0000","last_became_active":"2024-04-26T01:07:48.148023+0000","last_became_peered":"2024-04-26T01:07:48.148023+0000","last_unstale":"2024-04-26T01:09:42.309414+0000","last_undegraded":"2024-04-26T01:09:42.309414+0000","last_fullsized":"2024-04-26T01:09:42.309414+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:07.988439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:07.988439+0000","last_clean_scrub_stamp":"2024-04-26T01:09:07.988439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:14.276176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":23216,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,4,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1617920,"data_stored":1701891,"data_compressed":346059,"data_compressed_allocated":491520,"data_compressed_original":1413120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,8,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54908,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316932,"statfs":{"total":96636764160,"available":96580538368,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313244,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32760,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1617920,"data_stored":1701891,"data_compressed":346059,"data_compressed_allocated":491520,"data_compressed_original":1413120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313243,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313243,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":21,"seq":90194313243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313244,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,3,4,5,6,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313243,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28540,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343300,"statfs":{"total":96636764160,"available":96607539200,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":60,"seq":257698037779,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29232,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27710,"kb_avail":94342608,"statfs":{"total":96636764160,"available":96606830592,"internally_reserved":0,"allocated":1544192,"data_stored":1163345,"data_compressed":331394,"data_compressed_allocated":446464,"data_compressed_original":892928,"omap_allocated":1802,"internal_metadata":28375286},"hb_peers":[0,2,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313245,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[5,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313244,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":96,"seq":412316860427,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29120,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27582,"kb_avail":94342720,"statfs":{"total":96636764160,"available":96606945280,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":2023,"internal_metadata":28243993},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:09:47.403 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T01:09:47.404 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:09:47.708 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:09:47.709 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-26T01:09:47.727 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-04-26T01:09:46.018011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":538512,"kb_used_data":24428,"kb_used_omap":25,"kb_used_meta":513830,"kb_avail":1509410928,"statfs":{"total":1546188226560,"available":1545636790272,"internally_reserved":0,"allocated":25014272,"data_stored":19815646,"data_compressed":5371375,"data_compressed_allocated":7286784,"data_compressed_original":15433728,"omap_allocated":26158,"internal_metadata":526162386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.576032"},"pg_stats":[{"pgid":"1.0","version":"27'157","reported_seq":426,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.461652+0000","last_change":"2024-04-26T01:09:12.876970+0000","last_active":"2024-04-26T01:09:41.461652+0000","last_peered":"2024-04-26T01:09:41.461652+0000","last_clean":"2024-04-26T01:09:41.461652+0000","last_became_active":"2024-04-26T01:07:46.407301+0000","last_became_peered":"2024-04-26T01:07:46.407301+0000","last_unstale":"2024-04-26T01:09:41.461652+0000","last_undegraded":"2024-04-26T01:09:41.461652+0000","last_fullsized":"2024-04-26T01:09:41.461652+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'157","last_scrub_stamp":"2024-04-26T01:09:12.876525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:45.031420+0000","last_clean_scrub_stamp":"2024-04-26T01:09:12.876525+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:35.766266+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":225,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.461874+0000","last_change":"2024-04-26T01:09:08.691918+0000","last_active":"2024-04-26T01:09:41.461874+0000","last_peered":"2024-04-26T01:09:41.461874+0000","last_clean":"2024-04-26T01:09:41.461874+0000","last_became_active":"2024-04-26T01:07:48.144313+0000","last_became_peered":"2024-04-26T01:07:48.144313+0000","last_unstale":"2024-04-26T01:09:41.461874+0000","last_undegraded":"2024-04-26T01:09:41.461874+0000","last_fullsized":"2024-04-26T01:09:41.461874+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:08.691698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:08.691698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:29.117900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":71,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.726383+0000","last_change":"2024-04-26T01:09:10.395016+0000","last_active":"2024-04-26T01:09:41.726383+0000","last_peered":"2024-04-26T01:09:41.726383+0000","last_clean":"2024-04-26T01:09:41.726383+0000","last_became_active":"2024-04-26T01:09:10.394370+0000","last_became_peered":"2024-04-26T01:09:10.394370+0000","last_unstale":"2024-04-26T01:09:41.726383+0000","last_undegraded":"2024-04-26T01:09:41.726383+0000","last_fullsized":"2024-04-26T01:09:41.726383+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:08.921880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:08.921880+0000","last_clean_scrub_stamp":"2024-04-26T01:09:08.921880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:18.094815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":226,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.461666+0000","last_change":"2024-04-26T01:09:15.910993+0000","last_active":"2024-04-26T01:09:41.461666+0000","last_peered":"2024-04-26T01:09:41.461666+0000","last_clean":"2024-04-26T01:09:41.461666+0000","last_became_active":"2024-04-26T01:07:48.145991+0000","last_became_peered":"2024-04-26T01:07:48.145991+0000","last_unstale":"2024-04-26T01:09:41.461666+0000","last_undegraded":"2024-04-26T01:09:41.461666+0000","last_fullsized":"2024-04-26T01:09:41.461666+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:15.910744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:15.910744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:35.650953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":79,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.726383+0000","last_change":"2024-04-26T01:09:12.345740+0000","last_active":"2024-04-26T01:09:41.726383+0000","last_peered":"2024-04-26T01:09:41.726383+0000","last_clean":"2024-04-26T01:09:41.726383+0000","last_became_active":"2024-04-26T01:09:10.896103+0000","last_became_peered":"2024-04-26T01:09:10.896103+0000","last_unstale":"2024-04-26T01:09:41.726383+0000","last_undegraded":"2024-04-26T01:09:41.726383+0000","last_fullsized":"2024-04-26T01:09:41.726383+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:12.345508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:12.345508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:13.867577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":228,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.459378+0000","last_change":"2024-04-26T01:08:50.478667+0000","last_active":"2024-04-26T01:09:41.459378+0000","last_peered":"2024-04-26T01:09:41.459378+0000","last_clean":"2024-04-26T01:09:41.459378+0000","last_became_active":"2024-04-26T01:07:48.465232+0000","last_became_peered":"2024-04-26T01:07:48.465232+0000","last_unstale":"2024-04-26T01:09:41.459378+0000","last_undegraded":"2024-04-26T01:09:41.459378+0000","last_fullsized":"2024-04-26T01:09:41.459378+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:08:50.478571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:08:50.478571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:08.707444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":228,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:41.501932+0000","last_change":"2024-04-26T01:09:14.870357+0000","last_active":"2024-04-26T01:09:41.501932+0000","last_peered":"2024-04-26T01:09:41.501932+0000","last_clean":"2024-04-26T01:09:41.501932+0000","last_became_active":"2024-04-26T01:07:48.145376+0000","last_became_peered":"2024-04-26T01:07:48.145376+0000","last_unstale":"2024-04-26T01:09:41.501932+0000","last_undegraded":"2024-04-26T01:09:41.501932+0000","last_fullsized":"2024-04-26T01:09:41.501932+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:14.870031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:14.870031+0000","last_clean_scrub_stamp":"2024-04-26T01:09:14.870031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:26.187184+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":227,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:42.929264+0000","last_change":"2024-04-26T01:09:17.636007+0000","last_active":"2024-04-26T01:09:42.929264+0000","last_peered":"2024-04-26T01:09:42.929264+0000","last_clean":"2024-04-26T01:09:42.929264+0000","last_became_active":"2024-04-26T01:07:48.145512+0000","last_became_peered":"2024-04-26T01:07:48.145512+0000","last_unstale":"2024-04-26T01:09:42.929264+0000","last_undegraded":"2024-04-26T01:09:42.929264+0000","last_fullsized":"2024-04-26T01:09:42.929264+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:17.635797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:17.635797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:34.968710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":226,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-26T01:09:42.309414+0000","last_change":"2024-04-26T01:09:07.988749+0000","last_active":"2024-04-26T01:09:42.309414+0000","last_peered":"2024-04-26T01:09:42.309414+0000","last_clean":"2024-04-26T01:09:42.309414+0000","last_became_active":"2024-04-26T01:07:48.148023+0000","last_became_peered":"2024-04-26T01:07:48.148023+0000","last_unstale":"2024-04-26T01:09:42.309414+0000","last_undegraded":"2024-04-26T01:09:42.309414+0000","last_fullsized":"2024-04-26T01:09:42.309414+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:07.988439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:07.988439+0000","last_clean_scrub_stamp":"2024-04-26T01:09:07.988439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:14.276176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":23216,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,4,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1617920,"data_stored":1701891,"data_compressed":346059,"data_compressed_allocated":491520,"data_compressed_original":1413120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,8,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54908,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316932,"statfs":{"total":96636764160,"available":96580538368,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313244,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32760,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1617920,"data_stored":1701891,"data_compressed":346059,"data_compressed_allocated":491520,"data_compressed_original":1413120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313243,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313243,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":21,"seq":90194313243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313244,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,3,4,5,6,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313243,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28540,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343300,"statfs":{"total":96636764160,"available":96607539200,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":60,"seq":257698037779,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29232,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27710,"kb_avail":94342608,"statfs":{"total":96636764160,"available":96606830592,"internally_reserved":0,"allocated":1544192,"data_stored":1163345,"data_compressed":331394,"data_compressed_allocated":446464,"data_compressed_original":892928,"omap_allocated":1802,"internal_metadata":28375286},"hb_peers":[0,2,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313245,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[5,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313244,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32700,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":96,"seq":412316860427,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29120,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27582,"kb_avail":94342720,"statfs":{"total":96636764160,"available":96606945280,"internally_reserved":0,"allocated":1556480,"data_stored":1172963,"data_compressed":334451,"data_compressed_allocated":450560,"data_compressed_original":901120,"omap_allocated":2023,"internal_metadata":28243993},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:09:47.729 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T01:09:48.027 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:09:48.028 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":133,"fsid":"4f4f63a3-e6cc-48b9-8eb8-e5ea97ff4554","created":"2024-04-26T01:07:31.350368+0000","modified":"2024-04-26T01:09:47.740805+0000","last_up_change":"2024-04-26T01:09:03.085260+0000","last_in_change":"2024-04-26T01:09:09.126496+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:07:44.085247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T01:07:46.423491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"abed240a-e896-459b-8a43-cd1419cafbc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6801","nonce":188308042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6803","nonce":188308042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6807","nonce":188308042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":188308042},{"type":"v1","addr":"172.21.15.8:6805","nonce":188308042}]},"public_addr":"172.21.15.8:6801/188308042","cluster_addr":"172.21.15.8:6803/188308042","heartbeat_back_addr":"172.21.15.8:6807/188308042","heartbeat_front_addr":"172.21.15.8:6805/188308042","state":["exists","up"]},{"osd":1,"uuid":"831c717c-e124-4489-a44f-93db78c47deb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6819","nonce":3487767810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6823","nonce":3487767810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6831","nonce":3487767810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":3487767810},{"type":"v1","addr":"172.21.15.12:6828","nonce":3487767810}]},"public_addr":"172.21.15.12:6819/3487767810","cluster_addr":"172.21.15.12:6823/3487767810","heartbeat_back_addr":"172.21.15.12:6831/3487767810","heartbeat_front_addr":"172.21.15.12:6828/3487767810","state":["exists","up"]},{"osd":2,"uuid":"92a95165-ae34-4f29-a7f3-fe97afc26af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6803","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6805","nonce":3599623558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6807","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6809","nonce":3599623558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6815","nonce":3599623558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3599623558},{"type":"v1","addr":"172.21.15.116:6813","nonce":3599623558}]},"public_addr":"172.21.15.116:6805/3599623558","cluster_addr":"172.21.15.116:6809/3599623558","heartbeat_back_addr":"172.21.15.116:6815/3599623558","heartbeat_front_addr":"172.21.15.116:6813/3599623558","state":["exists","up"]},{"osd":3,"uuid":"e9d8f764-4951-4de4-92c7-682850852e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":79,"up_from":96,"up_thru":100,"down_at":80,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":115208799},{"type":"v1","addr":"172.21.15.164:6804","nonce":115208799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":115208799},{"type":"v1","addr":"172.21.15.164:6808","nonce":115208799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":115208799},{"type":"v1","addr":"172.21.15.164:6816","nonce":115208799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":115208799},{"type":"v1","addr":"172.21.15.164:6812","nonce":115208799}]},"public_addr":"172.21.15.164:6804/115208799","cluster_addr":"172.21.15.164:6808/115208799","heartbeat_back_addr":"172.21.15.164:6816/115208799","heartbeat_front_addr":"172.21.15.164:6812/115208799","state":["exists","up"]},{"osd":4,"uuid":"7f5e93f8-b1f6-41e7-8da9-b8e57d4259bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6825","nonce":425515720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6827","nonce":425515720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6831","nonce":425515720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":425515720},{"type":"v1","addr":"172.21.15.8:6829","nonce":425515720}]},"public_addr":"172.21.15.8:6825/425515720","cluster_addr":"172.21.15.8:6827/425515720","heartbeat_back_addr":"172.21.15.8:6831/425515720","heartbeat_front_addr":"172.21.15.8:6829/425515720","state":["exists","up"]},{"osd":5,"uuid":"f7b3a311-3bf6-4ae5-b7d2-dedbbe6c47ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6809","nonce":1641884791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6811","nonce":1641884791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6815","nonce":1641884791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1641884791},{"type":"v1","addr":"172.21.15.12:6813","nonce":1641884791}]},"public_addr":"172.21.15.12:6809/1641884791","cluster_addr":"172.21.15.12:6811/1641884791","heartbeat_back_addr":"172.21.15.12:6815/1641884791","heartbeat_front_addr":"172.21.15.12:6813/1641884791","state":["exists","up"]},{"osd":6,"uuid":"796d7212-13de-43b2-8aa4-4abc4fd210a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6817","nonce":156763037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6820","nonce":156763037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6825","nonce":156763037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6821","nonce":156763037},{"type":"v1","addr":"172.21.15.116:6823","nonce":156763037}]},"public_addr":"172.21.15.116:6817/156763037","cluster_addr":"172.21.15.116:6820/156763037","heartbeat_back_addr":"172.21.15.116:6825/156763037","heartbeat_front_addr":"172.21.15.116:6823/156763037","state":["exists","up"]},{"osd":7,"uuid":"ccb8cd0f-a0d6-44d7-86cc-809eff9687ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6826","nonce":2291946233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6829","nonce":2291946233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6833","nonce":2291946233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":2291946233},{"type":"v1","addr":"172.21.15.164:6831","nonce":2291946233}]},"public_addr":"172.21.15.164:6826/2291946233","cluster_addr":"172.21.15.164:6829/2291946233","heartbeat_back_addr":"172.21.15.164:6833/2291946233","heartbeat_front_addr":"172.21.15.164:6831/2291946233","state":["exists","up"]},{"osd":8,"uuid":"bf13e311-71b7-44ff-af1b-68d118935c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6809","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6811","nonce":2709504491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6813","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6815","nonce":2709504491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6821","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6823","nonce":2709504491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6817","nonce":2709504491},{"type":"v1","addr":"172.21.15.8:6819","nonce":2709504491}]},"public_addr":"172.21.15.8:6811/2709504491","cluster_addr":"172.21.15.8:6815/2709504491","heartbeat_back_addr":"172.21.15.8:6823/2709504491","heartbeat_front_addr":"172.21.15.8:6819/2709504491","state":["exists","up"]},{"osd":9,"uuid":"4864f46f-fd7b-48fc-908b-2564f7bda67b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6801","nonce":1155130892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6803","nonce":1155130892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6807","nonce":1155130892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1155130892},{"type":"v1","addr":"172.21.15.12:6805","nonce":1155130892}]},"public_addr":"172.21.15.12:6801/1155130892","cluster_addr":"172.21.15.12:6803/1155130892","heartbeat_back_addr":"172.21.15.12:6807/1155130892","heartbeat_front_addr":"172.21.15.12:6805/1155130892","state":["exists","up"]},{"osd":10,"uuid":"09cbb9ca-f609-409e-821b-93e695b87067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6822","nonce":1461831728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6827","nonce":1461831728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6831","nonce":1461831728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1461831728},{"type":"v1","addr":"172.21.15.116:6829","nonce":1461831728}]},"public_addr":"172.21.15.116:6822/1461831728","cluster_addr":"172.21.15.116:6827/1461831728","heartbeat_back_addr":"172.21.15.116:6831/1461831728","heartbeat_front_addr":"172.21.15.116:6829/1461831728","state":["exists","up"]},{"osd":11,"uuid":"9bfdf353-5174-40dd-bb3c-ab2dc850adad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":42,"up_from":60,"up_thru":30,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6819","nonce":838948348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6821","nonce":838948348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6825","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6827","nonce":838948348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":838948348},{"type":"v1","addr":"172.21.15.164:6823","nonce":838948348}]},"public_addr":"172.21.15.164:6819/838948348","cluster_addr":"172.21.15.164:6821/838948348","heartbeat_back_addr":"172.21.15.164:6827/838948348","heartbeat_front_addr":"172.21.15.164:6823/838948348","state":["exists","up"]},{"osd":12,"uuid":"a3e244d8-a765-467e-820c-caa5d1d97c9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6810","nonce":2082773890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6814","nonce":2082773890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6822","nonce":2082773890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":2082773890},{"type":"v1","addr":"172.21.15.8:6818","nonce":2082773890}]},"public_addr":"172.21.15.8:6810/2082773890","cluster_addr":"172.21.15.8:6814/2082773890","heartbeat_back_addr":"172.21.15.8:6822/2082773890","heartbeat_front_addr":"172.21.15.8:6818/2082773890","state":["exists","up"]},{"osd":13,"uuid":"383875c2-0a0c-4834-8beb-a6784eef1f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6817","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6818","nonce":3158583580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6822","nonce":3158583580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6829","nonce":3158583580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3158583580},{"type":"v1","addr":"172.21.15.12:6826","nonce":3158583580}]},"public_addr":"172.21.15.12:6818/3158583580","cluster_addr":"172.21.15.12:6822/3158583580","heartbeat_back_addr":"172.21.15.12:6829/3158583580","heartbeat_front_addr":"172.21.15.12:6826/3158583580","state":["exists","up"]},{"osd":14,"uuid":"30311d57-c625-4dff-98d5-1ba0a84c0ceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6801","nonce":2158650188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6804","nonce":2158650188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6812","nonce":2158650188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2158650188},{"type":"v1","addr":"172.21.15.116:6808","nonce":2158650188}]},"public_addr":"172.21.15.116:6801/2158650188","cluster_addr":"172.21.15.116:6804/2158650188","heartbeat_back_addr":"172.21.15.116:6812/2158650188","heartbeat_front_addr":"172.21.15.116:6808/2158650188","state":["exists","up"]},{"osd":15,"uuid":"1243478f-9908-460d-88d8-3c542d5258b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6805","nonce":2601606699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6809","nonce":2601606699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6817","nonce":2601606699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":2601606699},{"type":"v1","addr":"172.21.15.164:6813","nonce":2601606699}]},"public_addr":"172.21.15.164:6805/2601606699","cluster_addr":"172.21.15.164:6809/2601606699","heartbeat_back_addr":"172.21.15.164:6817/2601606699","heartbeat_front_addr":"172.21.15.164:6813/2601606699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:43.889257+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:46.000388+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:07.998091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:43.947103+0000","dead_epoch":80},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:10.476891+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:46.926348+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:47.952421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:11.399059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:51.155841+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:10.299960+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:58.178338+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:01.663819+0000","dead_epoch":43},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:09.972775+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:05.889679+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:08:47.436227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:09:05.698647+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":21,"length":1},{"begin":24,"length":2},{"begin":28,"length":3}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:09:49.046 INFO:tasks.ceph:Scrubbing osd.0 2024-04-26T01:09:49.046 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:49.222 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:49.222 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:09:49.223 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:49.239 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-26T01:09:49.540 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 0 to deep-scrub 2024-04-26T01:09:49.559 INFO:tasks.ceph:Scrubbing osd.4 2024-04-26T01:09:49.559 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:49.726 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:49.726 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:09:49.726 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:49.741 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-26T01:09:50.023 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 4 to deep-scrub 2024-04-26T01:09:50.041 INFO:tasks.ceph:Scrubbing osd.8 2024-04-26T01:09:50.042 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:50.192 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:50.192 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:09:50.192 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:50.208 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-04-26T01:09:50.505 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 8 to deep-scrub 2024-04-26T01:09:50.524 INFO:tasks.ceph:Scrubbing osd.12 2024-04-26T01:09:50.524 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:50.682 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:50.683 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:09:50.683 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:50.698 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-04-26T01:09:50.994 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 12 to deep-scrub 2024-04-26T01:09:51.011 INFO:tasks.ceph:Scrubbing osd.1 2024-04-26T01:09:51.012 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:51.176 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:51.177 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:09:51.177 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:51.195 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-26T01:09:51.480 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 1 to deep-scrub 2024-04-26T01:09:51.499 INFO:tasks.ceph:Scrubbing osd.5 2024-04-26T01:09:51.499 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:51.653 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:51.653 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:09:51.653 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:51.668 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-26T01:09:51.961 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 5 to deep-scrub 2024-04-26T01:09:51.978 INFO:tasks.ceph:Scrubbing osd.9 2024-04-26T01:09:51.978 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:52.148 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:52.148 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:09:52.149 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:52.165 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-04-26T01:09:52.459 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 9 to deep-scrub 2024-04-26T01:09:52.477 INFO:tasks.ceph:Scrubbing osd.13 2024-04-26T01:09:52.477 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:52.648 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:52.649 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:09:52.649 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:52.665 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-04-26T01:09:52.975 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 13 to deep-scrub 2024-04-26T01:09:52.995 INFO:tasks.ceph:Scrubbing osd.2 2024-04-26T01:09:52.995 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:53.155 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:53.155 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:09:53.156 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:53.171 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-26T01:09:53.466 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 2 to deep-scrub 2024-04-26T01:09:53.486 INFO:tasks.ceph:Scrubbing osd.6 2024-04-26T01:09:53.486 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:53.649 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:53.649 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:09:53.649 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:53.665 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-26T01:09:53.969 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 6 to deep-scrub 2024-04-26T01:09:53.987 INFO:tasks.ceph:Scrubbing osd.10 2024-04-26T01:09:53.987 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:54.151 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:54.152 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:09:54.152 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:54.167 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-04-26T01:09:54.463 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 10 to deep-scrub 2024-04-26T01:09:54.482 INFO:tasks.ceph:Scrubbing osd.14 2024-04-26T01:09:54.483 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:54.652 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:54.652 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:09:54.652 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:54.669 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-04-26T01:09:54.966 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 14 to deep-scrub 2024-04-26T01:09:54.984 INFO:tasks.ceph:Scrubbing osd.3 2024-04-26T01:09:54.984 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:55.147 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:55.147 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-04-26T01:09:55.147 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:55.162 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-26T01:09:55.446 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 3 to deep-scrub 2024-04-26T01:09:55.463 INFO:tasks.ceph:Scrubbing osd.7 2024-04-26T01:09:55.463 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:55.613 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:55.613 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:09:55.613 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:55.628 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-26T01:09:55.913 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 7 to deep-scrub 2024-04-26T01:09:55.931 INFO:tasks.ceph:Scrubbing osd.11 2024-04-26T01:09:55.931 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:56.102 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:56.102 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:09:56.102 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:56.119 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-04-26T01:09:56.418 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 11 to deep-scrub 2024-04-26T01:09:56.438 INFO:tasks.ceph:Scrubbing osd.15 2024-04-26T01:09:56.438 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:09:56.600 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-26T01:09:56.601 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:09:56.601 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-26T01:09:56.619 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-04-26T01:09:56.919 INFO:teuthology.orchestra.run.smithi008.stderr:instructed osd(s) 15 to deep-scrub 2024-04-26T01:09:56.937 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:09:57.242 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:09:57.243 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-26T01:09:57.261 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-04-26T01:09:56.021164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":538848,"kb_used_data":24800,"kb_used_omap":25,"kb_used_meta":513830,"kb_avail":1509410592,"statfs":{"total":1546188226560,"available":1545636446208,"internally_reserved":0,"allocated":25395200,"data_stored":20115064,"data_compressed":5466172,"data_compressed_allocated":7413760,"data_compressed_original":15687680,"omap_allocated":26158,"internal_metadata":526162386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.275030"},"pg_stats":[{"pgid":"1.0","version":"134'159","reported_seq":442,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-26T01:09:53.507924+0000","last_change":"2024-04-26T01:09:53.508056+0000","last_active":"2024-04-26T01:09:53.507924+0000","last_peered":"2024-04-26T01:09:53.507924+0000","last_clean":"2024-04-26T01:09:53.507924+0000","last_became_active":"2024-04-26T01:07:46.407301+0000","last_became_peered":"2024-04-26T01:07:46.407301+0000","last_unstale":"2024-04-26T01:09:53.507924+0000","last_undegraded":"2024-04-26T01:09:53.507924+0000","last_fullsized":"2024-04-26T01:09:53.507924+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'159","last_scrub_stamp":"2024-04-26T01:09:53.507703+0000","last_deep_scrub":"134'159","last_deep_scrub_stamp":"2024-04-26T01:09:53.507703+0000","last_clean_scrub_stamp":"2024-04-26T01:09:53.507703+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:09.296090+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":239,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-26T01:09:52.636272+0000","last_change":"2024-04-26T01:09:52.635284+0000","last_active":"2024-04-26T01:09:52.636272+0000","last_peered":"2024-04-26T01:09:52.636272+0000","last_clean":"2024-04-26T01:09:52.636272+0000","last_became_active":"2024-04-26T01:07:48.144313+0000","last_became_peered":"2024-04-26T01:07:48.144313+0000","last_unstale":"2024-04-26T01:09:52.636272+0000","last_undegraded":"2024-04-26T01:09:52.636272+0000","last_fullsized":"2024-04-26T01:09:52.636272+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:52.635046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:52.635046+0000","last_clean_scrub_stamp":"2024-04-26T01:09:52.635046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:04.003473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":75,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-26T01:09:52.805097+0000","last_change":"2024-04-26T01:09:10.395016+0000","last_active":"2024-04-26T01:09:52.805097+0000","last_peered":"2024-04-26T01:09:52.805097+0000","last_clean":"2024-04-26T01:09:52.805097+0000","last_became_active":"2024-04-26T01:09:10.394370+0000","last_became_peered":"2024-04-26T01:09:10.394370+0000","last_unstale":"2024-04-26T01:09:52.805097+0000","last_undegraded":"2024-04-26T01:09:52.805097+0000","last_fullsized":"2024-04-26T01:09:52.805097+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:08.921880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:08.921880+0000","last_clean_scrub_stamp":"2024-04-26T01:09:08.921880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:18.094815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":240,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-26T01:09:54.494599+0000","last_change":"2024-04-26T01:09:54.494599+0000","last_active":"2024-04-26T01:09:54.494599+0000","last_peered":"2024-04-26T01:09:54.494599+0000","last_clean":"2024-04-26T01:09:54.494599+0000","last_became_active":"2024-04-26T01:07:48.145991+0000","last_became_peered":"2024-04-26T01:07:48.145991+0000","last_unstale":"2024-04-26T01:09:54.494599+0000","last_undegraded":"2024-04-26T01:09:54.494599+0000","last_fullsized":"2024-04-26T01:09:54.494599+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:54.494333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:54.494333+0000","last_clean_scrub_stamp":"2024-04-26T01:09:54.494333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:21.218146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":83,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-26T01:09:52.805093+0000","last_change":"2024-04-26T01:09:12.345740+0000","last_active":"2024-04-26T01:09:52.805093+0000","last_peered":"2024-04-26T01:09:52.805093+0000","last_clean":"2024-04-26T01:09:52.805093+0000","last_became_active":"2024-04-26T01:09:10.896103+0000","last_became_peered":"2024-04-26T01:09:10.896103+0000","last_unstale":"2024-04-26T01:09:52.805093+0000","last_undegraded":"2024-04-26T01:09:52.805093+0000","last_fullsized":"2024-04-26T01:09:52.805093+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:12.345508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:07:47.115876+0000","last_clean_scrub_stamp":"2024-04-26T01:09:12.345508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:13.867577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-26T01:09:52.084762+0000","last_change":"2024-04-26T01:09:50.563413+0000","last_active":"2024-04-26T01:09:52.084762+0000","last_peered":"2024-04-26T01:09:52.084762+0000","last_clean":"2024-04-26T01:09:52.084762+0000","last_became_active":"2024-04-26T01:07:48.465232+0000","last_became_peered":"2024-04-26T01:07:48.465232+0000","last_unstale":"2024-04-26T01:09:52.084762+0000","last_undegraded":"2024-04-26T01:09:52.084762+0000","last_fullsized":"2024-04-26T01:09:52.084762+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:50.563278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:50.563278+0000","last_clean_scrub_stamp":"2024-04-26T01:09:50.563278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:00.235550+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":234,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-26T01:09:53.971974+0000","last_change":"2024-04-26T01:09:14.870357+0000","last_active":"2024-04-26T01:09:53.971974+0000","last_peered":"2024-04-26T01:09:53.971974+0000","last_clean":"2024-04-26T01:09:53.971974+0000","last_became_active":"2024-04-26T01:07:48.145376+0000","last_became_peered":"2024-04-26T01:07:48.145376+0000","last_unstale":"2024-04-26T01:09:53.971974+0000","last_undegraded":"2024-04-26T01:09:53.971974+0000","last_fullsized":"2024-04-26T01:09:53.971974+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:14.870031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:14.870031+0000","last_clean_scrub_stamp":"2024-04-26T01:09:14.870031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":240,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-26T01:09:52.148309+0000","last_change":"2024-04-26T01:09:51.448989+0000","last_active":"2024-04-26T01:09:52.148309+0000","last_peered":"2024-04-26T01:09:52.148309+0000","last_clean":"2024-04-26T01:09:52.148309+0000","last_became_active":"2024-04-26T01:07:48.145512+0000","last_became_peered":"2024-04-26T01:07:48.145512+0000","last_unstale":"2024-04-26T01:09:52.148309+0000","last_undegraded":"2024-04-26T01:09:52.148309+0000","last_fullsized":"2024-04-26T01:09:52.148309+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:51.448761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:51.448761+0000","last_clean_scrub_stamp":"2024-04-26T01:09:51.448761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:02.364926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":230,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-26T01:09:52.087195+0000","last_change":"2024-04-26T01:09:07.988749+0000","last_active":"2024-04-26T01:09:52.087195+0000","last_peered":"2024-04-26T01:09:52.087195+0000","last_clean":"2024-04-26T01:09:52.087195+0000","last_became_active":"2024-04-26T01:07:48.148023+0000","last_became_peered":"2024-04-26T01:07:48.148023+0000","last_unstale":"2024-04-26T01:09:52.087195+0000","last_undegraded":"2024-04-26T01:09:52.087195+0000","last_fullsized":"2024-04-26T01:09:52.087195+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:07.988439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:07.988439+0000","last_clean_scrub_stamp":"2024-04-26T01:09:07.988439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:10:14.276176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":23216,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":21,"seq":90194313247,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32724,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1581056,"data_stored":1192283,"data_compressed":340567,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,4,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32784,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339056,"statfs":{"total":96636764160,"available":96603193344,"internally_reserved":0,"allocated":1642496,"data_stored":1721211,"data_compressed":352175,"data_compressed_allocated":499712,"data_compressed_original":1429504,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32724,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1581056,"data_stored":1192283,"data_compressed":340567,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,8,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313245,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54932,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316908,"statfs":{"total":96636764160,"available":96580513792,"internally_reserved":0,"allocated":1581056,"data_stored":1192283,"data_compressed":340567,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32784,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339056,"statfs":{"total":96636764160,"available":96603193344,"internally_reserved":0,"allocated":1642496,"data_stored":1721211,"data_compressed":352175,"data_compressed_allocated":499712,"data_compressed_original":1429504,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32724,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1581056,"data_stored":1192283,"data_compressed":340567,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32724,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1581056,"data_stored":1192283,"data_compressed":340567,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32724,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1581056,"data_stored":1192283,"data_compressed":340567,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":21,"seq":90194313245,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32724,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1581056,"data_stored":1192283,"data_compressed":340567,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32724,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1581056,"data_stored":1192283,"data_compressed":340567,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,3,4,5,6,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28564,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343276,"statfs":{"total":96636764160,"available":96607514624,"internally_reserved":0,"allocated":1581056,"data_stored":1192283,"data_compressed":340567,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":60,"seq":257698037781,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29256,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27710,"kb_avail":94342584,"statfs":{"total":96636764160,"available":96606806016,"internally_reserved":0,"allocated":1568768,"data_stored":1182623,"data_compressed":337509,"data_compressed_allocated":454656,"data_compressed_original":909312,"omap_allocated":1802,"internal_metadata":28375286},"hb_peers":[0,2,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313247,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1581056,"data_stored":1192283,"data_compressed":340567,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[5,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32724,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1581056,"data_stored":1192283,"data_compressed":340567,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32724,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1581056,"data_stored":1192283,"data_compressed":340567,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":96,"seq":412316860429,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29128,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27582,"kb_avail":94342712,"statfs":{"total":96636764160,"available":96606937088,"internally_reserved":0,"allocated":1568768,"data_stored":1182623,"data_compressed":337509,"data_compressed_allocated":454656,"data_compressed_original":909312,"omap_allocated":2023,"internal_metadata":28243993},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:09:57.265 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-26T01:09:08.921880+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=9, tm_sec=8, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=9, tm_sec=48, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T01:09:57.265 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-26T01:09:12.345508+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=9, tm_sec=12, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=9, tm_sec=48, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T01:09:57.265 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-26T01:09:14.870031+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=9, tm_sec=14, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=9, tm_sec=48, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T01:09:57.266 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-26T01:09:07.988439+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=9, tm_sec=7, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=9, tm_sec=48, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T01:09:57.266 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-26T01:10:17.267 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:10:17.586 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-26T01:10:17.587 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-26T01:10:17.602 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-04-26T01:10:16.141260+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":539572,"kb_used_data":25520,"kb_used_omap":25,"kb_used_meta":513830,"kb_avail":1509409868,"statfs":{"total":1546188226560,"available":1545635704832,"internally_reserved":0,"allocated":26132480,"data_stored":20696014,"data_compressed":5649881,"data_compressed_allocated":7659520,"data_compressed_original":16179200,"omap_allocated":26158,"internal_metadata":526162386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.038291"},"pg_stats":[{"pgid":"1.0","version":"134'159","reported_seq":450,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-26T01:10:13.112252+0000","last_change":"2024-04-26T01:09:53.508056+0000","last_active":"2024-04-26T01:10:13.112252+0000","last_peered":"2024-04-26T01:10:13.112252+0000","last_clean":"2024-04-26T01:10:13.112252+0000","last_became_active":"2024-04-26T01:07:46.407301+0000","last_became_peered":"2024-04-26T01:07:46.407301+0000","last_unstale":"2024-04-26T01:10:13.112252+0000","last_undegraded":"2024-04-26T01:10:13.112252+0000","last_fullsized":"2024-04-26T01:10:13.112252+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'159","last_scrub_stamp":"2024-04-26T01:09:53.507703+0000","last_deep_scrub":"134'159","last_deep_scrub_stamp":"2024-04-26T01:09:53.507703+0000","last_clean_scrub_stamp":"2024-04-26T01:09:53.507703+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:09.296090+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":247,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-26T01:10:13.909808+0000","last_change":"2024-04-26T01:09:52.635284+0000","last_active":"2024-04-26T01:10:13.909808+0000","last_peered":"2024-04-26T01:10:13.909808+0000","last_clean":"2024-04-26T01:10:13.909808+0000","last_became_active":"2024-04-26T01:07:48.144313+0000","last_became_peered":"2024-04-26T01:07:48.144313+0000","last_unstale":"2024-04-26T01:10:13.909808+0000","last_undegraded":"2024-04-26T01:10:13.909808+0000","last_fullsized":"2024-04-26T01:10:13.909808+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:52.635046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:52.635046+0000","last_clean_scrub_stamp":"2024-04-26T01:09:52.635046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:04.003473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":90,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-26T01:10:10.628018+0000","last_change":"2024-04-26T01:09:56.064085+0000","last_active":"2024-04-26T01:10:10.628018+0000","last_peered":"2024-04-26T01:10:10.628018+0000","last_clean":"2024-04-26T01:10:10.628018+0000","last_became_active":"2024-04-26T01:09:10.394370+0000","last_became_peered":"2024-04-26T01:09:10.394370+0000","last_unstale":"2024-04-26T01:10:10.628018+0000","last_undegraded":"2024-04-26T01:10:10.628018+0000","last_fullsized":"2024-04-26T01:10:10.628018+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:56.063782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:56.063782+0000","last_clean_scrub_stamp":"2024-04-26T01:09:56.063782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:04.615535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":248,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-26T01:10:13.112227+0000","last_change":"2024-04-26T01:09:54.494599+0000","last_active":"2024-04-26T01:10:13.112227+0000","last_peered":"2024-04-26T01:10:13.112227+0000","last_clean":"2024-04-26T01:10:13.112227+0000","last_became_active":"2024-04-26T01:07:48.145991+0000","last_became_peered":"2024-04-26T01:07:48.145991+0000","last_unstale":"2024-04-26T01:10:13.112227+0000","last_undegraded":"2024-04-26T01:10:13.112227+0000","last_fullsized":"2024-04-26T01:10:13.112227+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:54.494333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:54.494333+0000","last_clean_scrub_stamp":"2024-04-26T01:09:54.494333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:21.218146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":98,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-26T01:10:10.628061+0000","last_change":"2024-04-26T01:09:57.070821+0000","last_active":"2024-04-26T01:10:10.628061+0000","last_peered":"2024-04-26T01:10:10.628061+0000","last_clean":"2024-04-26T01:10:10.628061+0000","last_became_active":"2024-04-26T01:09:10.896103+0000","last_became_peered":"2024-04-26T01:09:10.896103+0000","last_unstale":"2024-04-26T01:10:10.628061+0000","last_undegraded":"2024-04-26T01:10:10.628061+0000","last_fullsized":"2024-04-26T01:10:10.628061+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:57.070591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:57.070591+0000","last_clean_scrub_stamp":"2024-04-26T01:09:57.070591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:07.711561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":250,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-26T01:10:13.483542+0000","last_change":"2024-04-26T01:09:50.563413+0000","last_active":"2024-04-26T01:10:13.483542+0000","last_peered":"2024-04-26T01:10:13.483542+0000","last_clean":"2024-04-26T01:10:13.483542+0000","last_became_active":"2024-04-26T01:07:48.465232+0000","last_became_peered":"2024-04-26T01:07:48.465232+0000","last_unstale":"2024-04-26T01:10:13.483542+0000","last_undegraded":"2024-04-26T01:10:13.483542+0000","last_fullsized":"2024-04-26T01:10:13.483542+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:50.563278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:50.563278+0000","last_clean_scrub_stamp":"2024-04-26T01:09:50.563278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:00.235550+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":250,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-26T01:10:13.485077+0000","last_change":"2024-04-26T01:09:54.831064+0000","last_active":"2024-04-26T01:10:13.485077+0000","last_peered":"2024-04-26T01:10:13.485077+0000","last_clean":"2024-04-26T01:10:13.485077+0000","last_became_active":"2024-04-26T01:07:48.145376+0000","last_became_peered":"2024-04-26T01:07:48.145376+0000","last_unstale":"2024-04-26T01:10:13.485077+0000","last_undegraded":"2024-04-26T01:10:13.485077+0000","last_fullsized":"2024-04-26T01:10:13.485077+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:54.830823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:54.830823+0000","last_clean_scrub_stamp":"2024-04-26T01:09:54.830823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:08.588500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":249,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-26T01:10:13.483614+0000","last_change":"2024-04-26T01:09:51.448989+0000","last_active":"2024-04-26T01:10:13.483614+0000","last_peered":"2024-04-26T01:10:13.483614+0000","last_clean":"2024-04-26T01:10:13.483614+0000","last_became_active":"2024-04-26T01:07:48.145512+0000","last_became_peered":"2024-04-26T01:07:48.145512+0000","last_unstale":"2024-04-26T01:10:13.483614+0000","last_undegraded":"2024-04-26T01:10:13.483614+0000","last_fullsized":"2024-04-26T01:10:13.483614+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:51.448761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:51.448761+0000","last_clean_scrub_stamp":"2024-04-26T01:09:51.448761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:02.364926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":248,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-26T01:10:13.483484+0000","last_change":"2024-04-26T01:09:57.860870+0000","last_active":"2024-04-26T01:10:13.483484+0000","last_peered":"2024-04-26T01:10:13.483484+0000","last_clean":"2024-04-26T01:10:13.483484+0000","last_became_active":"2024-04-26T01:07:48.148023+0000","last_became_peered":"2024-04-26T01:07:48.148023+0000","last_unstale":"2024-04-26T01:10:13.483484+0000","last_undegraded":"2024-04-26T01:10:13.483484+0000","last_fullsized":"2024-04-26T01:10:13.483484+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:09:57.860501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:09:57.860501+0000","last_clean_scrub_stamp":"2024-04-26T01:09:57.860501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:18.727110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":23216,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":21,"seq":90194313251,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32776,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339064,"statfs":{"total":96636764160,"available":96603201536,"internally_reserved":0,"allocated":1630208,"data_stored":1231023,"data_compressed":352815,"data_compressed_allocated":475136,"data_compressed_original":950272,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,4,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32832,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339008,"statfs":{"total":96636764160,"available":96603144192,"internally_reserved":0,"allocated":1691648,"data_stored":1759951,"data_compressed":364423,"data_compressed_allocated":516096,"data_compressed_original":1462272,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32772,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339068,"statfs":{"total":96636764160,"available":96603205632,"internally_reserved":0,"allocated":1630208,"data_stored":1231023,"data_compressed":352815,"data_compressed_allocated":475136,"data_compressed_original":950272,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,8,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313249,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54968,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316872,"statfs":{"total":96636764160,"available":96580476928,"internally_reserved":0,"allocated":1617920,"data_stored":1221313,"data_compressed":349753,"data_compressed_allocated":471040,"data_compressed_original":942080,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313250,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32832,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339008,"statfs":{"total":96636764160,"available":96603144192,"internally_reserved":0,"allocated":1691648,"data_stored":1759951,"data_compressed":364423,"data_compressed_allocated":516096,"data_compressed_original":1462272,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313251,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32772,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339068,"statfs":{"total":96636764160,"available":96603205632,"internally_reserved":0,"allocated":1630208,"data_stored":1231023,"data_compressed":352815,"data_compressed_allocated":475136,"data_compressed_original":950272,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313249,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32772,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339068,"statfs":{"total":96636764160,"available":96603205632,"internally_reserved":0,"allocated":1630208,"data_stored":1231023,"data_compressed":352815,"data_compressed_allocated":475136,"data_compressed_original":950272,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313249,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32772,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339068,"statfs":{"total":96636764160,"available":96603205632,"internally_reserved":0,"allocated":1630208,"data_stored":1231023,"data_compressed":352815,"data_compressed_allocated":475136,"data_compressed_original":950272,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":21,"seq":90194313249,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32772,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339068,"statfs":{"total":96636764160,"available":96603205632,"internally_reserved":0,"allocated":1630208,"data_stored":1231023,"data_compressed":352815,"data_compressed_allocated":475136,"data_compressed_original":950272,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313250,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32772,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339068,"statfs":{"total":96636764160,"available":96603205632,"internally_reserved":0,"allocated":1630208,"data_stored":1231023,"data_compressed":352815,"data_compressed_allocated":475136,"data_compressed_original":950272,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,3,4,5,6,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313249,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28600,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343240,"statfs":{"total":96636764160,"available":96607477760,"internally_reserved":0,"allocated":1617920,"data_stored":1221313,"data_compressed":349753,"data_compressed_allocated":471040,"data_compressed_original":942080,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":60,"seq":257698037785,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29292,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27710,"kb_avail":94342548,"statfs":{"total":96636764160,"available":96606769152,"internally_reserved":0,"allocated":1605632,"data_stored":1211653,"data_compressed":346688,"data_compressed_allocated":466944,"data_compressed_original":933888,"omap_allocated":1802,"internal_metadata":28375286},"hb_peers":[0,2,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313251,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36932,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334908,"statfs":{"total":96636764160,"available":96598945792,"internally_reserved":0,"allocated":1630208,"data_stored":1231023,"data_compressed":352815,"data_compressed_allocated":475136,"data_compressed_original":950272,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[5,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313250,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32760,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1617920,"data_stored":1221313,"data_compressed":349753,"data_compressed_allocated":471040,"data_compressed_original":942080,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32772,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339068,"statfs":{"total":96636764160,"available":96603205632,"internally_reserved":0,"allocated":1630208,"data_stored":1231023,"data_compressed":352815,"data_compressed_allocated":475136,"data_compressed_original":950272,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":96,"seq":412316860433,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29176,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":27582,"kb_avail":94342664,"statfs":{"total":96636764160,"available":96606887936,"internally_reserved":0,"allocated":1617920,"data_stored":1221313,"data_compressed":349753,"data_compressed_allocated":471040,"data_compressed_original":942080,"omap_allocated":2023,"internal_metadata":28243993},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":11608,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:10:17.604 DEBUG:teuthology.orchestra.run.smithi008:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-26T01:10:17.923 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-26T01:10:17.923 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-26T01:10:17.923 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-26T01:10:17.923 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:18.015 INFO:tasks.ceph.osd.0:Stopped 2024-04-26T01:10:18.015 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-26T01:10:18.016 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:18.096 INFO:tasks.ceph.osd.4:Stopped 2024-04-26T01:10:18.097 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-04-26T01:10:18.097 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:18.198 INFO:tasks.ceph.osd.8:Stopped 2024-04-26T01:10:18.198 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-04-26T01:10:18.198 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:18.271 INFO:tasks.ceph.osd.12:Stopped 2024-04-26T01:10:18.271 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-26T01:10:18.271 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:18.367 INFO:tasks.ceph.osd.1:Stopped 2024-04-26T01:10:18.368 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-26T01:10:18.368 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:18.466 INFO:tasks.ceph.osd.5:Stopped 2024-04-26T01:10:18.467 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-04-26T01:10:18.467 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:19.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-04-26T01:10:24.568 INFO:tasks.ceph.osd.9:Stopped 2024-04-26T01:10:24.569 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-04-26T01:10:24.569 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:24.647 INFO:tasks.ceph.osd.13:Stopped 2024-04-26T01:10:24.648 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-26T01:10:24.648 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:24.723 INFO:tasks.ceph.osd.2:Stopped 2024-04-26T01:10:24.724 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-26T01:10:24.724 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:24.811 INFO:tasks.ceph.osd.6:Stopped 2024-04-26T01:10:24.811 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-04-26T01:10:24.811 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:24.904 INFO:tasks.ceph.osd.10:Stopped 2024-04-26T01:10:24.904 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-04-26T01:10:24.904 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:24.991 INFO:tasks.ceph.osd.14:Stopped 2024-04-26T01:10:24.992 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-26T01:10:24.992 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:25.077 INFO:tasks.ceph.osd.3:Stopped 2024-04-26T01:10:25.077 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-26T01:10:25.077 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:25.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.14 is failed for ~0s 2024-04-26T01:10:25.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-04-26T01:10:25.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-04-26T01:10:31.179 INFO:tasks.ceph.osd.7:Stopped 2024-04-26T01:10:31.179 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-04-26T01:10:31.179 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:31.230 INFO:tasks.ceph.osd.11:Stopped 2024-04-26T01:10:31.230 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-04-26T01:10:31.230 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:31.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-04-26T01:10:31.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.14 has been restored 2024-04-26T01:10:31.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-04-26T01:10:36.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-26T01:10:36.041+0000 7f848a81e640 -1 calc_pg_upmaps abort due to max <= 0 2024-04-26T01:10:37.087 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.15 is failed for ~0s 2024-04-26T01:10:37.087 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-04-26T01:10:37.332 INFO:tasks.ceph.osd.15:Stopped 2024-04-26T01:10:37.332 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-26T01:10:37.332 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-26T01:10:37.332 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:37.361 INFO:tasks.ceph.mgr.y:Stopped 2024-04-26T01:10:37.361 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-26T01:10:37.361 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:37.393 INFO:tasks.ceph.mgr.x:Stopped 2024-04-26T01:10:37.394 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-26T01:10:37.394 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T01:10:37.394 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:37.420 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T01:10:37.420 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T01:10:37.420 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:37.441 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T01:10:37.441 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T01:10:37.442 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:10:37.466 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T01:10:37.467 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-26T01:10:37.467 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-04-26T01:10:37.487 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi008.front.sepia.ceph.com 2024-04-26T01:10:37.487 DEBUG:teuthology.orchestra.run.smithi008:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-26T01:10:37.714 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi008.front.sepia.ceph.com 2024-04-26T01:10:37.714 DEBUG:teuthology.orchestra.run.smithi008:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-26T01:10:37.932 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi008.front.sepia.ceph.com 2024-04-26T01:10:37.932 DEBUG:teuthology.orchestra.run.smithi008:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-04-26T01:10:38.142 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi008.front.sepia.ceph.com 2024-04-26T01:10:38.142 DEBUG:teuthology.orchestra.run.smithi008:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-04-26T01:10:38.316 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-26T01:10:38.316 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-26T01:10:38.470 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-26T01:10:38.471 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-26T01:10:38.626 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-26T01:10:38.627 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-04-26T01:10:38.803 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-26T01:10:38.803 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-04-26T01:10:38.950 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-26T01:10:38.951 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-26T01:10:39.372 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-26T01:10:39.372 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-26T01:10:39.565 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-26T01:10:39.565 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-04-26T01:10:39.723 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-26T01:10:39.723 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-04-26T01:10:39.898 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi164.front.sepia.ceph.com 2024-04-26T01:10:39.898 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-26T01:10:40.042 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi164.front.sepia.ceph.com 2024-04-26T01:10:40.042 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-26T01:10:40.215 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi164.front.sepia.ceph.com 2024-04-26T01:10:40.215 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-04-26T01:10:40.349 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi164.front.sepia.ceph.com 2024-04-26T01:10:40.349 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-04-26T01:10:40.466 INFO:tasks.ceph:Archiving mon data... 2024-04-26T01:10:40.468 DEBUG:teuthology.misc:Transferring archived files from smithi008:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/data/mon.a.tgz 2024-04-26T01:10:40.468 DEBUG:teuthology.orchestra.run.smithi008:> mktemp 2024-04-26T01:10:40.476 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.0OnovDncBz 2024-04-26T01:10:40.476 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0OnovDncBz 2024-04-26T01:10:40.626 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.0OnovDncBz 2024-04-26T01:10:40.688 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.0OnovDncBz is 425KB 2024-04-26T01:10:40.750 DEBUG:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.0OnovDncBz 2024-04-26T01:10:40.755 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/data/mon.b.tgz 2024-04-26T01:10:40.756 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-04-26T01:10:40.762 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.t3SaDpTzGA 2024-04-26T01:10:40.762 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.t3SaDpTzGA 2024-04-26T01:10:40.955 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.t3SaDpTzGA 2024-04-26T01:10:41.023 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.t3SaDpTzGA is 598KB 2024-04-26T01:10:41.089 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.t3SaDpTzGA 2024-04-26T01:10:41.094 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/data/mon.c.tgz 2024-04-26T01:10:41.094 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-04-26T01:10:41.100 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.cH2IOgBXDI 2024-04-26T01:10:41.101 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.cH2IOgBXDI 2024-04-26T01:10:41.284 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.cH2IOgBXDI 2024-04-26T01:10:41.345 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.cH2IOgBXDI is 598KB 2024-04-26T01:10:41.408 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.cH2IOgBXDI 2024-04-26T01:10:41.413 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-26T01:10:41.413 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T01:10:41.416 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T01:10:41.418 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T01:10:41.459 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T01:10:41.479 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T01:10:41.485 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T01:10:41.518 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T01:10:41.523 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T01:10:41.523 INFO:tasks.ceph:Archiving crash dumps... 2024-04-26T01:10:41.525 DEBUG:teuthology.misc:Transferring archived files from smithi008:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/remote/smithi008/crash 2024-04-26T01:10:41.526 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T01:10:41.541 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/remote/smithi012/crash 2024-04-26T01:10:41.542 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T01:10:41.556 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/remote/smithi116/crash 2024-04-26T01:10:41.557 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T01:10:41.574 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/remote/smithi164/crash 2024-04-26T01:10:41.575 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T01:10:41.589 INFO:tasks.ceph:Compressing logs... 2024-04-26T01:10:41.589 DEBUG:teuthology.orchestra.run.smithi008:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T01:10:41.592 DEBUG:teuthology.orchestra.run.smithi012:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T01:10:41.604 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T01:10:41.612 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29532.log 2024-04-26T01:10:41.613 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33514.log 2024-04-26T01:10:41.613 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29532.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.29532.log.gz 2024-04-26T01:10:41.613 INFO:teuthology.orchestra.run.smithi008.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34760.log 2024-04-26T01:10:41.614 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32751.log 2024-04-26T01:10:41.614 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33514.log.gz 2024-04-26T01:10:41.615 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33841.log 2024-04-26T01:10:41.615 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34760.log: /var/log/ceph/ceph-client.admin.32751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32314.log 2024-04-26T01:10:41.615 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32751.log.gz 2024-04-26T01:10:41.615 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34760.log.gz 2024-04-26T01:10:41.615 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31642.log 2024-04-26T01:10:41.616 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33841.log.gz 2024-04-26T01:10:41.616 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32314.log: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.32314.log.gz 2024-04-26T01:10:41.617 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-26T01:10:41.617 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35335.log 2024-04-26T01:10:41.617 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31642.log.gz 2024-04-26T01:10:41.617 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34790.log 2024-04-26T01:10:41.617 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.admin.35335.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.35335.log.gz -5 2024-04-26T01:10:41.617 INFO:teuthology.orchestra.run.smithi008.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29480.log 2024-04-26T01:10:41.621 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33071.log 2024-04-26T01:10:41.621 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34790.log.gz 2024-04-26T01:10:41.621 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29480.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-26T01:10:41.622 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29480.log.gz 2024-04-26T01:10:41.622 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32243.log 2024-04-26T01:10:41.622 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33071.log.gz 2024-04-26T01:10:41.622 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20120.log 2024-04-26T01:10:41.622 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33859.log 2024-04-26T01:10:41.622 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32243.log.gz 2024-04-26T01:10:41.622 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph.tmp-client.admin.20120.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20120.log.gz 2024-04-26T01:10:41.622 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34249.log 2024-04-26T01:10:41.622 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35543.log 2024-04-26T01:10:41.622 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33859.log.gz 2024-04-26T01:10:41.622 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34249.log.gz 2024-04-26T01:10:41.622 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31668.log 2024-04-26T01:10:41.622 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.35543.log: /var/log/ceph/ceph-client.admin.33358.log 2024-04-26T01:10:41.623 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35543.log.gz 2024-04-26T01:10:41.623 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31668.log: 24.3% -- replaced with /var/log/ceph/ceph-client.admin.31668.log.gz 2024-04-26T01:10:41.623 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34786.log 2024-04-26T01:10:41.623 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33358.log.gz 2024-04-26T01:10:41.623 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34270.log 2024-04-26T01:10:41.623 INFO:teuthology.orchestra.run.smithi008.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-26T01:10:41.623 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34786.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.28641.log -- replaced with /var/log/ceph/ceph-client.admin.34786.log.gz 2024-04-26T01:10:41.623 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-26T01:10:41.624 DEBUG:teuthology.orchestra.run.smithi164:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T01:10:41.628 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-26T01:10:41.628 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-26T01:10:41.628 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-26T01:10:41.628 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-26T01:10:41.628 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-04-26T01:10:41.629 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-04-26T01:10:41.629 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-26T01:10:41.629 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-osd.13.log: /var/log/ceph/ceph-mon.b.log: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-26T01:10:41.629 INFO:teuthology.orchestra.run.smithi012.stderr: 86.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-26T01:10:41.645 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-26T01:10:41.646 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-26T01:10:41.646 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-04-26T01:10:41.646 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-26T01:10:41.646 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-26T01:10:41.646 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.14.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-26T01:10:41.646 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-04-26T01:10:41.646 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-osd.10.log: 86.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-26T01:10:41.648 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-26T01:10:41.676 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.35548.log 2024-04-26T01:10:41.677 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30402.log 2024-04-26T01:10:41.677 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/objectstore_tool.35548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24950.log 2024-04-26T01:10:41.677 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/objectstore_tool.30402.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.37070.log 2024-04-26T01:10:41.677 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24878.log 2024-04-26T01:10:41.677 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.24950.log: /var/log/ceph/bluestore_tool.37070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24950.log.gz 2024-04-26T01:10:41.677 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.34807.log 2024-04-26T01:10:41.677 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.24878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25174.log 2024-04-26T01:10:41.677 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24878.log.gz 2024-04-26T01:10:41.677 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/objectstore_tool.34807.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-26T01:10:41.677 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.25174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25149.log 2024-04-26T01:10:41.678 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25174.log.gz 2024-04-26T01:10:41.678 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24926.log 2024-04-26T01:10:41.678 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.25149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25198.log 2024-04-26T01:10:41.678 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25149.log.gz 2024-04-26T01:10:41.678 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.24926.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.31929.log 2024-04-26T01:10:41.678 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24926.log.gz 2024-04-26T01:10:41.678 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.25198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25101.log 2024-04-26T01:10:41.678 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25198.log.gz 2024-04-26T01:10:41.678 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/bluestore_tool.31929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24998.log 2024-04-26T01:10:41.678 INFO:teuthology.orchestra.run.smithi164.stderr: 45.6% -- replaced with /var/log/ceph/bluestore_tool.31929.log.gz 2024-04-26T01:10:41.678 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.29661.log 2024-04-26T01:10:41.678 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.25101.log: /var/log/ceph/ceph-client.admin.24998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25101.log.gz 2024-04-26T01:10:41.679 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25053.log 2024-04-26T01:10:41.679 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.24998.log.gz 2024-04-26T01:10:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/objectstore_tool.29661.log: 93.7% -- replaced with /var/log/ceph/bluestore_tool.37070.log.gz 2024-04-26T01:10:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-26T01:10:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.33289.log 2024-04-26T01:10:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.25053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25053.log.gz/var/log/ceph/ceph-osd.3.log: 2024-04-26T01:10:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.34066.log 2024-04-26T01:10:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.32305.log 2024-04-26T01:10:41.682 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/objectstore_tool.33289.log: /var/log/ceph/objectstore_tool.34066.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-04-26T01:10:41.698 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/bluestore_tool.32305.log: 93.5% -- replaced with /var/log/ceph/objectstore_tool.34807.log.gz 2024-04-26T01:10:41.699 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/objectstore_tool.30402.log.gz 2024-04-26T01:10:41.699 INFO:teuthology.orchestra.run.smithi164.stderr: 93.3% -- replaced with /var/log/ceph/objectstore_tool.35548.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.0.28812.log 2024-04-26T01:10:41.699 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-04-26T01:10:41.699 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.28920.log 2024-04-26T01:10:41.699 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.0.28812.log: 71.5% -- replaced with /var/log/ceph/ceph-client.0.28812.log.gz 2024-04-26T01:10:41.699 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24854.log 2024-04-26T01:10:41.700 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/objectstore_tool.28920.log: 93.5% -- replaced with /var/log/ceph/objectstore_tool.29661.log.gz 2024-04-26T01:10:41.700 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-04-26T01:10:41.700 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.24854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24854.log.gz 2024-04-26T01:10:41.700 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25125.log 2024-04-26T01:10:41.700 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.15.log: 93.7% -- replaced with /var/log/ceph/objectstore_tool.34066.log.gz 2024-04-26T01:10:41.700 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25222.log 2024-04-26T01:10:41.700 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.25125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25125.log.gz 2024-04-26T01:10:41.700 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-26T01:10:41.700 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.25222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25222.log.gz 2024-04-26T01:10:41.700 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.31143.log 2024-04-26T01:10:41.700 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mgr.x.log: 93.5% -- replaced with /var/log/ceph/bluestore_tool.32305.log.gz 2024-04-26T01:10:41.700 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25077.log 2024-04-26T01:10:41.716 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/objectstore_tool.31143.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.36298.log 2024-04-26T01:10:41.716 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.25077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25077.log.gz 2024-04-26T01:10:41.716 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.36678.log 2024-04-26T01:10:41.716 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/bluestore_tool.36298.log: 45.6% -- replaced with /var/log/ceph/bluestore_tool.36298.log.gz 2024-04-26T01:10:41.716 INFO:teuthology.orchestra.run.smithi164.stderr: 93.2% -- replaced with /var/log/ceph/objectstore_tool.31143.log.gz 2024-04-26T01:10:41.716 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24974.log 2024-04-26T01:10:41.726 INFO:teuthology.orchestra.run.smithi012.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-26T01:10:41.734 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28921.log 0.0% 2024-04-26T01:10:41.734 INFO:teuthology.orchestra.run.smithi008.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34270.log.gz 2024-04-26T01:10:41.734 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36037.log/var/log/ceph/ceph-client.admin.28641.log: 2024-04-26T01:10:41.734 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28641.log.gz 2024-04-26T01:10:41.734 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28921.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28921.log.gz 2024-04-26T01:10:41.734 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35179.log 2024-04-26T01:10:41.734 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.36037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34212.log 2024-04-26T01:10:41.734 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36037.log.gz 2024-04-26T01:10:41.735 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32041.log 2024-04-26T01:10:41.735 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35179.log.gz 2024-04-26T01:10:41.735 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34212.log.gz 2024-04-26T01:10:41.735 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33488.log 2024-04-26T01:10:41.735 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32200.log 2024-04-26T01:10:41.735 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32041.log.gz 2024-04-26T01:10:41.735 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30752.log 2024-04-26T01:10:41.735 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33488.log.gz 2024-04-26T01:10:41.735 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34789.log 2024-04-26T01:10:41.735 INFO:teuthology.orchestra.run.smithi008.stderr: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.32200.log.gz 2024-04-26T01:10:41.735 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35309.log 2024-04-26T01:10:41.735 INFO:teuthology.orchestra.run.smithi008.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.30752.log.gz 2024-04-26T01:10:41.735 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35127.log 2024-04-26T01:10:41.736 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34789.log.gz 2024-04-26T01:10:41.736 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34683.log 2024-04-26T01:10:41.736 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35309.log.gz 2024-04-26T01:10:41.736 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36063.log 2024-04-26T01:10:41.736 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35127.log.gz 2024-04-26T01:10:41.736 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32993.log 2024-04-26T01:10:41.736 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34683.log.gz 2024-04-26T01:10:41.736 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.36063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36063.log.gz 2024-04-26T01:10:41.736 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-04-26T01:10:41.736 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34319.log 2024-04-26T01:10:41.736 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32993.log.gz 2024-04-26T01:10:41.736 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30131.log 2024-04-26T01:10:41.736 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-osd.8.log: /var/log/ceph/ceph-client.admin.34319.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.34319.log.gz /var/log/ceph/ceph-client.admin.31904.log 2024-04-26T01:10:41.736 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-26T01:10:41.737 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30131.log.gz 2024-04-26T01:10:41.737 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34248.log 2024-04-26T01:10:41.737 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31904.log.gz 2024-04-26T01:10:41.737 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36011.log 2024-04-26T01:10:41.737 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30207.log 2024-04-26T01:10:41.737 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34248.log: /var/log/ceph/ceph-client.admin.36011.log: 26.0% -- replaced with /var/log/ceph/ceph-client.admin.34248.log.gz 2024-04-26T01:10:41.737 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36011.log.gz 2024-04-26T01:10:41.737 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33254.log 2024-04-26T01:10:41.737 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34314.log 2024-04-26T01:10:41.737 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30207.log: /var/log/ceph/ceph-client.admin.33254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33254.log.gz 2024-04-26T01:10:41.737 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29506.log 2024-04-26T01:10:41.737 INFO:teuthology.orchestra.run.smithi008.stderr: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.30207.log.gz 2024-04-26T01:10:41.737 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34261.log 2024-04-26T01:10:41.737 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34314.log.gz 2024-04-26T01:10:41.738 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29506.log.gz 2024-04-26T01:10:41.738 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30845.log 2024-04-26T01:10:41.738 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.34261.log: -- /var/log/ceph/ceph-client.admin.31717.log 2024-04-26T01:10:41.738 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34261.log.gz 2024-04-26T01:10:41.738 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30845.log: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.30845.log.gz 2024-04-26T01:10:41.738 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34235.log 2024-04-26T01:10:41.738 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-26T01:10:41.738 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31717.log.gz 2024-04-26T01:10:41.738 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34235.log.gz 2024-04-26T01:10:41.738 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31954.log 2024-04-26T01:10:41.738 INFO:teuthology.orchestra.run.smithi008.stderr:gzip/var/log/ceph/ceph-mon.a.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33332.log 2024-04-26T01:10:41.738 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31954.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31954.log.gz 2024-04-26T01:10:41.738 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34136.log 2024-04-26T01:10:41.738 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35907.log 2024-04-26T01:10:41.739 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33332.log.gz 2024-04-26T01:10:41.739 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34136.log.gzgzip 2024-04-26T01:10:41.739 INFO:teuthology.orchestra.run.smithi008.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32941.log 2024-04-26T01:10:41.739 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35907.log.gz 2024-04-26T01:10:41.739 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31407.log 2024-04-26T01:10:41.739 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35699.log 2024-04-26T01:10:41.739 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32941.log.gz 2024-04-26T01:10:41.739 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31407.log.gz 2024-04-26T01:10:41.739 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30290.log 2024-04-26T01:10:41.739 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35699.log.gz 2024-04-26T01:10:41.739 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33280.log 2024-04-26T01:10:41.739 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32175.log 2024-04-26T01:10:41.739 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30290.log: /var/log/ceph/ceph-client.admin.33280.log: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.30290.log.gz 2024-04-26T01:10:41.739 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30609.log 2024-04-26T01:10:41.740 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33280.log.gz 2024-04-26T01:10:41.740 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32175.log.gz 2024-04-26T01:10:41.740 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34246.log 2024-04-26T01:10:41.740 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33123.log 2024-04-26T01:10:41.740 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30609.log: /var/log/ceph/ceph-client.admin.34246.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30609.log.gz 2024-04-26T01:10:41.740 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34246.log.gz 2024-04-26T01:10:41.740 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35855.log 2024-04-26T01:10:41.740 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34697.log 2024-04-26T01:10:41.740 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33123.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33123.log.gz 2024-04-26T01:10:41.740 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35855.log.gz 2024-04-26T01:10:41.740 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36091.log 2024-04-26T01:10:41.740 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34697.log.gz 2024-04-26T01:10:41.740 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28520.log 2024-04-26T01:10:41.741 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32097.log 2024-04-26T01:10:41.741 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.36091.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36091.log.gz 2024-04-26T01:10:41.741 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28520.log.gz 2024-04-26T01:10:41.741 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31592.log 2024-04-26T01:10:41.741 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32097.log.gz 2024-04-26T01:10:41.741 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35361.log 2024-04-26T01:10:41.741 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32149.log 2024-04-26T01:10:41.741 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31592.log.gz 2024-04-26T01:10:41.741 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35361.log.gz 2024-04-26T01:10:41.741 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28635.log 2024-04-26T01:10:41.741 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32149.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32149.log.gz 2024-04-26T01:10:41.741 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32411.log 2024-04-26T01:10:41.741 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33019.log 2024-04-26T01:10:41.741 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28635.log: 0.0%/var/log/ceph/ceph-client.admin.32411.log: -- replaced with /var/log/ceph/ceph-client.admin.28635.log.gz 2024-04-26T01:10:41.741 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29063.log 2024-04-26T01:10:41.742 INFO:teuthology.orchestra.run.smithi008.stderr: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.32411.log.gz 2024-04-26T01:10:41.742 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33019.log.gz 2024-04-26T01:10:41.742 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30372.log 2024-04-26T01:10:41.742 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33774.log 2024-04-26T01:10:41.742 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29063.log.gz 2024-04-26T01:10:41.742 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30372.log.gz 2024-04-26T01:10:41.742 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31275.log 2024-04-26T01:10:41.742 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33774.log.gz 2024-04-26T01:10:41.742 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33540.log 2024-04-26T01:10:41.742 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33202.log 2024-04-26T01:10:41.742 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31275.log: /var/log/ceph/ceph-client.admin.33540.log: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.31275.log.gz 2024-04-26T01:10:41.742 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.33540.log.gz 2024-04-26T01:10:41.743 INFO:teuthology.orchestra.run.smithi008.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33384.log 2024-04-26T01:10:41.743 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33202.log.gz 2024-04-26T01:10:41.743 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30690.log 2024-04-26T01:10:41.743 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33842.log 2024-04-26T01:10:41.743 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33384.log.gz 2024-04-26T01:10:41.743 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30690.log.gz 2024-04-26T01:10:41.743 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35205.log 2024-04-26T01:10:41.743 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33842.log.gz 2024-04-26T01:10:41.743 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31046.log 2024-04-26T01:10:41.743 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33857.log 2024-04-26T01:10:41.743 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35205.log.gz 2024-04-26T01:10:41.743 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31046.log.gz 2024-04-26T01:10:41.743 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33858.log 2024-04-26T01:10:41.743 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35647.log 2024-04-26T01:10:41.743 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33857.log.gz 2024-04-26T01:10:41.744 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33858.log.gz 2024-04-26T01:10:41.744 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33592.log 2024-04-26T01:10:41.744 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35725.log 2024-04-26T01:10:41.744 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35647.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35647.log.gz 2024-04-26T01:10:41.744 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33592.log.gz 2024-04-26T01:10:41.744 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30926.log 2024-04-26T01:10:41.744 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35725.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35725.log.gz 2024-04-26T01:10:41.744 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34633.log 2024-04-26T01:10:41.744 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34280.log 2024-04-26T01:10:41.744 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30926.log: /var/log/ceph/ceph-client.admin.34633.log: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.30926.log.gz 2024-04-26T01:10:41.744 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34633.log.gz 2024-04-26T01:10:41.744 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28488.log 2024-04-26T01:10:41.744 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33696.log 2024-04-26T01:10:41.744 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34280.log.gz 2024-04-26T01:10:41.745 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28488.log.gz 2024-04-26T01:10:41.745 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29049.log 2024-04-26T01:10:41.745 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35959.log 2024-04-26T01:10:41.745 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33696.log.gz 2024-04-26T01:10:41.745 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28670.log 2024-04-26T01:10:41.745 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29049.log.gz 2024-04-26T01:10:41.745 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35959.log.gz 2024-04-26T01:10:41.745 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31779.log 2024-04-26T01:10:41.745 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29134.log 2024-04-26T01:10:41.745 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28670.log: /var/log/ceph/ceph-client.admin.31779.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28670.log.gz 2024-04-26T01:10:41.745 INFO:teuthology.orchestra.run.smithi008.stderr:gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.31779.log.gz --verbose 2024-04-26T01:10:41.745 INFO:teuthology.orchestra.run.smithi008.stderr: -- /var/log/ceph/ceph-client.admin.30449.log 2024-04-26T01:10:41.745 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29134.log.gz 2024-04-26T01:10:41.745 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31461.log 2024-04-26T01:10:41.745 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29064.log 2024-04-26T01:10:41.746 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30449.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30449.log.gz 2024-04-26T01:10:41.746 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31461.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.31461.log.gz -- 2024-04-26T01:10:41.746 INFO:teuthology.orchestra.run.smithi008.stderr: /var/log/ceph/ceph-client.admin.35569.log 2024-04-26T01:10:41.746 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29064.log.gz 2024-04-26T01:10:41.746 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33462.log 2024-04-26T01:10:41.746 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35933.log 2024-04-26T01:10:41.746 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35569.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35569.log.gz 2024-04-26T01:10:41.746 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33462.log.gz 2024-04-26T01:10:41.746 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31471.log 2024-04-26T01:10:41.746 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35933.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35933.log.gz 2024-04-26T01:10:41.746 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30262.log 2024-04-26T01:10:41.746 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29087.log 2024-04-26T01:10:41.747 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31471.log: /var/log/ceph/ceph-client.admin.30262.log: 24.9% -- replaced with /var/log/ceph/ceph-client.admin.31471.log.gz 2024-04-26T01:10:41.747 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33748.log 2024-04-26T01:10:41.747 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30262.log.gz 2024-04-26T01:10:41.747 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29087.log.gz 2024-04-26T01:10:41.747 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34170.log 2024-04-26T01:10:41.747 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33849.log 2024-04-26T01:10:41.747 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33748.log.gz 2024-04-26T01:10:41.747 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28663.log 2024-04-26T01:10:41.747 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34170.log.gz 2024-04-26T01:10:41.747 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33849.log.gz 2024-04-26T01:10:41.747 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32492.log 2024-04-26T01:10:41.747 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33436.log 2024-04-26T01:10:41.747 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28663.log.gz 2024-04-26T01:10:41.748 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32492.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32492.log.gz 2024-04-26T01:10:41.748 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31252.log 2024-04-26T01:10:41.748 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35257.log 2024-04-26T01:10:41.748 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33436.log.gz 2024-04-26T01:10:41.748 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34637.log 2024-04-26T01:10:41.748 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31252.log.gz 2024-04-26T01:10:41.748 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35257.log.gz 2024-04-26T01:10:41.748 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29640.log 2024-04-26T01:10:41.748 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34268.log 2024-04-26T01:10:41.748 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34637.log.gz 2024-04-26T01:10:41.748 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29640.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.30345.log 2024-04-26T01:10:41.748 INFO:teuthology.orchestra.run.smithi008.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29640.log.gz 2024-04-26T01:10:41.748 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34268.log.gz 2024-04-26T01:10:41.748 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33306.log 2024-04-26T01:10:41.749 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29558.log 2024-04-26T01:10:41.749 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30345.log: 0.0%/var/log/ceph/ceph-client.admin.33306.log: -- replaced with /var/log/ceph/ceph-client.admin.30345.log.gz 2024-04-26T01:10:41.749 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33306.log.gz 2024-04-26T01:10:41.749 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28579.log 2024-04-26T01:10:41.749 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34725.log 2024-04-26T01:10:41.749 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29558.log.gz 2024-04-26T01:10:41.749 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28579.log.gz 2024-04-26T01:10:41.749 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29876.log 2024-04-26T01:10:41.749 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34725.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34725.log.gz 2024-04-26T01:10:41.749 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31355.log 2024-04-26T01:10:41.749 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29043.log 2024-04-26T01:10:41.749 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29876.log: /var/log/ceph/ceph-client.admin.31355.log: 25.8% -- replaced with /var/log/ceph/ceph-client.admin.29876.log.gz 2024-04-26T01:10:41.749 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35621.log 2024-04-26T01:10:41.749 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31355.log.gz 2024-04-26T01:10:41.750 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29043.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.29993.log 2024-04-26T01:10:41.750 INFO:teuthology.orchestra.run.smithi008.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29043.log.gz 2024-04-26T01:10:41.750 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34814.log 2024-04-26T01:10:41.750 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35621.log.gz 2024-04-26T01:10:41.750 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34661.log 2024-04-26T01:10:41.750 INFO:teuthology.orchestra.run.smithi008.stderr: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.29993.log.gz 2024-04-26T01:10:41.750 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34814.log.gz 2024-04-26T01:10:41.750 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33852.log 2024-04-26T01:10:41.750 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24892.log 2024-04-26T01:10:41.750 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34661.log: /var/log/ceph/ceph-client.admin.33852.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33852.log.gz -- replaced with /var/log/ceph/ceph-client.admin.34661.log.gz 2024-04-26T01:10:41.750 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-26T01:10:41.750 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34190.log 2024-04-26T01:10:41.750 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35985.log 2024-04-26T01:10:41.751 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.24892.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24892.log.gz 2024-04-26T01:10:41.751 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34190.log.gz 2024-04-26T01:10:41.751 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31828.log 2024-04-26T01:10:41.751 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33860.log 2024-04-26T01:10:41.751 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35985.log.gz 2024-04-26T01:10:41.751 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31828.log: 25.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.31828.log.gz -5 2024-04-26T01:10:41.751 INFO:teuthology.orchestra.run.smithi008.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29857.log 2024-04-26T01:10:41.751 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31987.log 2024-04-26T01:10:41.751 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33860.log.gz 2024-04-26T01:10:41.751 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29857.log.gz 2024-04-26T01:10:41.751 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33644.log 2024-04-26T01:10:41.751 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35439.log 2024-04-26T01:10:41.751 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31987.log.gz 2024-04-26T01:10:41.751 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33644.log.gz 2024-04-26T01:10:41.751 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30819.log 2024-04-26T01:10:41.752 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33566.log 2024-04-26T01:10:41.752 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35439.log.gz 2024-04-26T01:10:41.752 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30819.log.gz 2024-04-26T01:10:41.752 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-26T01:10:41.752 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33176.log 2024-04-26T01:10:41.752 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33566.log.gz 2024-04-26T01:10:41.752 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35101.log 2024-04-26T01:10:41.752 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.33176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35049.log 2024-04-26T01:10:41.752 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33176.log.gz 2024-04-26T01:10:41.752 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35465.log 2024-04-26T01:10:41.752 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35101.log.gz 2024-04-26T01:10:41.752 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35049.log.gz 2024-04-26T01:10:41.752 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32538.log 2024-04-26T01:10:41.752 INFO:teuthology.orchestra.run.smithi008.stderr: 86.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-26T01:10:41.753 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5/var/log/ceph/ceph-client.admin.35465.log: --verbose -- /var/log/ceph/ceph-client.admin.33824.log 2024-04-26T01:10:41.753 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35465.log.gz 2024-04-26T01:10:41.753 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32780.log 2024-04-26T01:10:41.753 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32538.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32538.log.gz 2024-04-26T01:10:41.753 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28667.log 2024-04-26T01:10:41.753 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33824.log.gz 2024-04-26T01:10:41.753 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31222.log 2024-04-26T01:10:41.753 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32780.log.gz 2024-04-26T01:10:41.753 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28667.log.gz 2024-04-26T01:10:41.753 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29072.log 2024-04-26T01:10:41.753 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29091.log 2024-04-26T01:10:41.753 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31222.log.gz 2024-04-26T01:10:41.753 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29072.log.gz 2024-04-26T01:10:41.753 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30524.log 2024-04-26T01:10:41.754 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29045.log 2024-04-26T01:10:41.754 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29091.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29091.log.gz 2024-04-26T01:10:41.754 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28552.log 2024-04-26T01:10:41.754 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30524.log.gz 2024-04-26T01:10:41.754 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32858.log 2024-04-26T01:10:41.754 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29045.log.gz 2024-04-26T01:10:41.754 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32967.log 2024-04-26T01:10:41.754 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28552.log.gz 2024-04-26T01:10:41.754 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32884.log 2024-04-26T01:10:41.754 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32858.log.gz 2024-04-26T01:10:41.754 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32967.log.gz 2024-04-26T01:10:41.754 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28646.log 2024-04-26T01:10:41.754 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33825.log 2024-04-26T01:10:41.754 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32884.log.gz 2024-04-26T01:10:41.755 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33834.log 2024-04-26T01:10:41.755 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28646.log.gz 2024-04-26T01:10:41.755 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33825.log.gz 2024-04-26T01:10:41.755 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33150.log 2024-04-26T01:10:41.755 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31753.log 2024-04-26T01:10:41.755 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33834.log.gz 2024-04-26T01:10:41.755 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34722.log 2024-04-26T01:10:41.755 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33150.log.gz 2024-04-26T01:10:41.755 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31753.log.gz 2024-04-26T01:10:41.755 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32415.log 2024-04-26T01:10:41.755 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33836.log 2024-04-26T01:10:41.755 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34722.log.gz 2024-04-26T01:10:41.755 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35231.log 2024-04-26T01:10:41.755 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32415.log.gz 2024-04-26T01:10:41.756 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33836.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33836.log.gz 2024-04-26T01:10:41.756 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33826.log 2024-04-26T01:10:41.756 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28462.log 2024-04-26T01:10:41.756 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35231.log.gz 2024-04-26T01:10:41.756 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29682.log 2024-04-26T01:10:41.756 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33826.log.gz 2024-04-26T01:10:41.756 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28462.log.gz 2024-04-26T01:10:41.756 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28669.log 2024-04-26T01:10:41.756 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36115.log 2024-04-26T01:10:41.756 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29682.log.gz 2024-04-26T01:10:41.756 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35413.log 2024-04-26T01:10:41.756 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28669.log.gz 2024-04-26T01:10:41.756 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.36115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36115.log.gz 2024-04-26T01:10:41.756 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35075.log 2024-04-26T01:10:41.757 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35153.log 2024-04-26T01:10:41.757 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35413.log.gz 2024-04-26T01:10:41.757 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29678.log 2024-04-26T01:10:41.757 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35075.log.gz 2024-04-26T01:10:41.757 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35153.log.gz 2024-04-26T01:10:41.757 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31563.log 2024-04-26T01:10:41.757 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31457.log 2024-04-26T01:10:41.757 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29678.log: 24.9% -- replaced with /var/log/ceph/ceph-client.admin.29678.log.gz 2024-04-26T01:10:41.757 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28664.log 2024-04-26T01:10:41.757 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31563.log.gz 2024-04-26T01:10:41.757 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31457.log.gz 2024-04-26T01:10:41.757 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35283.log 2024-04-26T01:10:41.757 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35881.log 2024-04-26T01:10:41.758 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28664.log.gz 2024-04-26T01:10:41.758 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33722.log 2024-04-26T01:10:41.758 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35283.log.gz 2024-04-26T01:10:41.758 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35881.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.35881.log.gz 2024-04-26T01:10:41.758 INFO:teuthology.orchestra.run.smithi008.stderr: -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-26T01:10:41.758 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31617.log 2024-04-26T01:10:41.758 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33722.log.gz 2024-04-26T01:10:41.758 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28665.log 2024-04-26T01:10:41.758 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28653.log 2024-04-26T01:10:41.758 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31617.log.gz 2024-04-26T01:10:41.758 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35491.log 2024-04-26T01:10:41.758 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28665.log.gz 2024-04-26T01:10:41.758 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29073.log 2024-04-26T01:10:41.758 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28653.log.gz 2024-04-26T01:10:41.759 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35491.log: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-26T01:10:41.759 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30901.log 2024-04-26T01:10:41.759 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35491.log.gz 2024-04-26T01:10:41.759 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.29073.log: /var/log/ceph/ceph-client.admin.29055.log 2024-04-26T01:10:41.759 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29073.log.gz 2024-04-26T01:10:41.759 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30901.log.gz 2024-04-26T01:10:41.759 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35673.log 2024-04-26T01:10:41.759 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32643.log 2024-04-26T01:10:41.759 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29055.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29055.log.gz 2024-04-26T01:10:41.759 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34682.log 2024-04-26T01:10:41.759 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35673.log.gz 2024-04-26T01:10:41.759 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31116.log 2024-04-26T01:10:41.759 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32643.log.gz 2024-04-26T01:10:41.759 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34682.log.gz 2024-04-26T01:10:41.760 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34669.log 2024-04-26T01:10:41.760 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31116.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-04-26T01:10:41.760 INFO:teuthology.orchestra.run.smithi008.stderr: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.31116.log.gz 2024-04-26T01:10:41.760 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31014.log 2024-04-26T01:10:41.760 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34669.log.gz 2024-04-26T01:10:41.760 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33850.log 2024-04-26T01:10:41.760 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31014.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.31014.log.gz -5 2024-04-26T01:10:41.760 INFO:teuthology.orchestra.run.smithi008.stderr: --verbose -- /var/log/ceph/ceph-client.admin.33830.log 2024-04-26T01:10:41.760 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30635.log 2024-04-26T01:10:41.760 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33850.log.gz 2024-04-26T01:10:41.760 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34772.log 2024-04-26T01:10:41.760 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33830.log.gz 2024-04-26T01:10:41.760 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35829.log 2024-04-26T01:10:41.760 INFO:teuthology.orchestra.run.smithi008.stderr: 24.9% -- replaced with /var/log/ceph/ceph-client.admin.30635.log.gz 2024-04-26T01:10:41.761 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35777.log 2024-04-26T01:10:41.761 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34772.log.gz 2024-04-26T01:10:41.761 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35829.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.35829.log.gz -5 2024-04-26T01:10:41.761 INFO:teuthology.orchestra.run.smithi008.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29967.log 2024-04-26T01:10:41.761 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29454.log 2024-04-26T01:10:41.761 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35777.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35777.log.gz 2024-04-26T01:10:41.761 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29083.log 2024-04-26T01:10:41.761 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29967.log.gz 2024-04-26T01:10:41.761 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29056.log 2024-04-26T01:10:41.761 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29454.log.gz 2024-04-26T01:10:41.761 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31248.log 2024-04-26T01:10:41.761 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29083.log.gz 2024-04-26T01:10:41.762 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30092.log 2024-04-26T01:10:41.762 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29056.log.gz 2024-04-26T01:10:41.762 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33045.log 2024-04-26T01:10:41.762 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31248.log.gz 2024-04-26T01:10:41.762 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35595.log 2024-04-26T01:10:41.762 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30092.log: /var/log/ceph/ceph-client.admin.33045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32310.log 2024-04-26T01:10:41.762 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33045.log.gz 2024-04-26T01:10:41.762 INFO:teuthology.orchestra.run.smithi008.stderr: 25.0% -- replaced with /var/log/ceph/ceph-client.admin.30092.log.gz 2024-04-26T01:10:41.762 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28666.log 2024-04-26T01:10:41.762 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35595.log.gz 2024-04-26T01:10:41.762 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31803.log 2024-04-26T01:10:41.762 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32310.log.gz 2024-04-26T01:10:41.763 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28638.log 2024-04-26T01:10:41.763 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28666.log.gz 2024-04-26T01:10:41.763 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33835.log 2024-04-26T01:10:41.763 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31803.log: 0.0%/var/log/ceph/ceph-client.admin.28638.log: -- replaced with /var/log/ceph/ceph-client.admin.31803.log.gz 2024-04-26T01:10:41.763 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31117.log 2024-04-26T01:10:41.763 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28638.log.gz 2024-04-26T01:10:41.763 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32806.log 2024-04-26T01:10:41.763 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33835.log.gz 2024-04-26T01:10:41.763 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29610.log 2024-04-26T01:10:41.763 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31117.log.gz 2024-04-26T01:10:41.763 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32806.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-26T01:10:41.763 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32806.log.gz 2024-04-26T01:10:41.763 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34233.log 2024-04-26T01:10:41.763 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29610.log.gz 2024-04-26T01:10:41.764 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32609.log 2024-04-26T01:10:41.764 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34233.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34233.log.gz 2024-04-26T01:10:41.764 INFO:teuthology.orchestra.run.smithi008.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34257.log 2024-04-26T01:10:41.764 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33097.log 2024-04-26T01:10:41.764 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32609.log.gz 2024-04-26T01:10:41.764 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30554.log 2024-04-26T01:10:41.764 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34257.log: /var/log/ceph/ceph-client.admin.33097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30407.log 2024-04-26T01:10:41.764 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34257.log.gz 2024-04-26T01:10:41.764 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33097.log.gz 2024-04-26T01:10:41.764 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30554.log: 24.9% -- replaced with /var/log/ceph/ceph-client.admin.30554.log.gz 2024-04-26T01:10:41.764 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29791.log 2024-04-26T01:10:41.764 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.30407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32651.log 2024-04-26T01:10:41.764 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30407.log.gz 2024-04-26T01:10:41.764 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29791.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29791.log.gz -5 2024-04-26T01:10:41.765 INFO:teuthology.orchestra.run.smithi008.stderr: --verbose -- /var/log/ceph/ceph-client.admin.25021.log 2024-04-26T01:10:41.765 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35751.log 2024-04-26T01:10:41.765 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32651.log: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.32651.log.gz 2024-04-26T01:10:41.765 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.25021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25021.log.gz 2024-04-26T01:10:41.765 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28651.log 2024-04-26T01:10:41.765 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35517.log 2024-04-26T01:10:41.765 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35751.log.gz 2024-04-26T01:10:41.765 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32832.log 2024-04-26T01:10:41.765 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28651.log.gz 2024-04-26T01:10:41.765 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28634.log 2024-04-26T01:10:41.765 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35517.log.gz 2024-04-26T01:10:41.765 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25130.log 2024-04-26T01:10:41.765 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32832.log: /var/log/ceph/ceph-client.admin.28634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32832.log.gz 2024-04-26T01:10:41.766 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28634.log.gz 2024-04-26T01:10:41.766 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29584.log 2024-04-26T01:10:41.766 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.25130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28668.log 2024-04-26T01:10:41.766 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25130.log.gz 2024-04-26T01:10:41.766 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35387.log 2024-04-26T01:10:41.766 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29584.log.gz 2024-04-26T01:10:41.766 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32915.log 2024-04-26T01:10:41.766 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29012.log 2024-04-26T01:10:41.766 INFO:teuthology.orchestra.run.smithi008.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28668.log.gz 2024-04-26T01:10:41.766 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35387.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35387.log.gz 2024-04-26T01:10:41.766 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32915.log.gz 2024-04-26T01:10:41.766 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28658.log 2024-04-26T01:10:41.766 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35803.log 2024-04-26T01:10:41.766 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29012.log.gz 2024-04-26T01:10:41.767 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29683.log 2024-04-26T01:10:41.767 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28658.log.gz 2024-04-26T01:10:41.767 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.35803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29154.log 2024-04-26T01:10:41.767 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35803.log.gz 2024-04-26T01:10:41.767 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34751.log 2024-04-26T01:10:41.767 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29683.log: /var/log/ceph/ceph-client.admin.29154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29683.log.gz 2024-04-26T01:10:41.767 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29154.log.gz 2024-04-26T01:10:41.767 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33618.log 2024-04-26T01:10:41.767 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.34751.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.34751.log.gz 2024-04-26T01:10:41.767 INFO:teuthology.orchestra.run.smithi008.stderr: --verbose -- /var/log/ceph/ceph-client.admin.31991.log 2024-04-26T01:10:41.767 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31196.log 2024-04-26T01:10:41.767 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33618.log.gz 2024-04-26T01:10:41.767 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.31991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29862.log 2024-04-26T01:10:41.768 INFO:teuthology.orchestra.run.smithi008.stderr: 23.9% -- replaced with /var/log/ceph/ceph-client.admin.31991.log.gz 2024-04-26T01:10:41.768 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28437.log 2024-04-26T01:10:41.768 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29862.log: /var/log/ceph/ceph-client.admin.31196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29862.log.gz 2024-04-26T01:10:41.768 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31196.log.gz 2024-04-26T01:10:41.768 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32715.log 2024-04-26T01:10:41.768 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.28437.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.28437.log.gz /var/log/ceph/ceph-client.admin.29997.log 2024-04-26T01:10:41.768 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-26T01:10:41.768 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33410.log 2024-04-26T01:10:41.768 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32715.log.gz 2024-04-26T01:10:41.768 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.29997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33228.log 2024-04-26T01:10:41.768 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29997.log.gz 2024-04-26T01:10:41.768 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32123.log 2024-04-26T01:10:41.768 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33410.log.gz 2024-04-26T01:10:41.768 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33228.log.gz 2024-04-26T01:10:41.769 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33670.log 2024-04-26T01:10:41.769 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.32123.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32123.log.gz 2024-04-26T01:10:41.769 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph-client.admin.33670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33670.log.gz 2024-04-26T01:10:41.771 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/bluestore_tool.36678.log: 93.5% -- replaced with /var/log/ceph/bluestore_tool.36678.log.gz 2024-04-26T01:10:41.772 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.31933.log 2024-04-26T01:10:41.772 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.24974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24974.log.gz 2024-04-26T01:10:41.772 INFO:teuthology.orchestra.run.smithi164.stderr: 94.4% -- replaced with /var/log/ceph/objectstore_tool.33289.log.gz 2024-04-26T01:10:41.772 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24902.log 2024-04-26T01:10:41.772 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/bluestore_tool.31933.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.32677.log 2024-04-26T01:10:41.772 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.24902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24902.log.gz 2024-04-26T01:10:41.772 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25029.log 2024-04-26T01:10:41.772 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/bluestore_tool.32677.log: 93.7% -- replaced with /var/log/ceph/bluestore_tool.31933.log.gz 2024-04-26T01:10:41.772 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.36302.log 2024-04-26T01:10:41.772 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.25029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25029.log.gz 2024-04-26T01:10:41.772 INFO:teuthology.orchestra.run.smithi164.stderr: 91.8% -- replaced with /var/log/ceph/bluestore_tool.32677.log.gz 2024-04-26T01:10:41.772 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/bluestore_tool.36302.log: 94.3% -- replaced with /var/log/ceph/objectstore_tool.28920.log.gz 2024-04-26T01:10:41.772 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/bluestore_tool.36302.log.gz 2024-04-26T01:10:41.774 INFO:teuthology.orchestra.run.smithi164.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-26T01:10:41.782 INFO:teuthology.orchestra.run.smithi116.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-26T01:10:41.923 INFO:teuthology.orchestra.run.smithi008.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-26T01:10:41.930 INFO:teuthology.orchestra.run.smithi116.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-04-26T01:10:42.061 INFO:teuthology.orchestra.run.smithi164.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-26T01:10:42.070 INFO:teuthology.orchestra.run.smithi012.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-04-26T01:10:42.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.15 has been restored 2024-04-26T01:10:42.096 INFO:teuthology.orchestra.run.smithi008.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-04-26T01:10:42.108 INFO:teuthology.orchestra.run.smithi116.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-04-26T01:10:42.327 INFO:teuthology.orchestra.run.smithi164.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-26T01:10:42.340 INFO:teuthology.orchestra.run.smithi164.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-04-26T01:10:42.391 INFO:teuthology.orchestra.run.smithi012.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-26T01:10:42.419 INFO:teuthology.orchestra.run.smithi008.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-04-26T01:10:42.424 INFO:teuthology.orchestra.run.smithi164.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-04-26T01:10:42.424 INFO:teuthology.orchestra.run.smithi116.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-26T01:10:42.425 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-04-26T01:10:42.425 INFO:teuthology.orchestra.run.smithi164.stderr:real 0m0.786s 2024-04-26T01:10:42.425 INFO:teuthology.orchestra.run.smithi164.stderr:user 0m2.719s 2024-04-26T01:10:42.426 INFO:teuthology.orchestra.run.smithi164.stderr:sys 0m0.129s 2024-04-26T01:10:42.456 INFO:teuthology.orchestra.run.smithi012.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-26T01:10:42.464 INFO:teuthology.orchestra.run.smithi116.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-26T01:10:42.466 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-04-26T01:10:42.466 INFO:teuthology.orchestra.run.smithi116.stderr:real 0m0.845s 2024-04-26T01:10:42.466 INFO:teuthology.orchestra.run.smithi116.stderr:user 0m2.465s 2024-04-26T01:10:42.466 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m0.066s 2024-04-26T01:10:42.497 INFO:teuthology.orchestra.run.smithi008.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-26T01:10:42.597 INFO:teuthology.orchestra.run.smithi012.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-04-26T01:10:42.599 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-04-26T01:10:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:real 0m0.993s 2024-04-26T01:10:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:user 0m3.017s 2024-04-26T01:10:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:sys 0m0.132s 2024-04-26T01:10:42.628 INFO:teuthology.orchestra.run.smithi008.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-26T01:10:42.630 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-26T01:10:42.630 INFO:teuthology.orchestra.run.smithi008.stderr:real 0m1.037s 2024-04-26T01:10:42.630 INFO:teuthology.orchestra.run.smithi008.stderr:user 0m3.418s 2024-04-26T01:10:42.630 INFO:teuthology.orchestra.run.smithi008.stderr:sys 0m0.158s 2024-04-26T01:10:42.631 INFO:tasks.ceph:Archiving logs... 2024-04-26T01:10:42.631 DEBUG:teuthology.misc:Transferring archived files from smithi008:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/remote/smithi008/log 2024-04-26T01:10:42.632 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T01:10:43.279 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/remote/smithi012/log 2024-04-26T01:10:43.280 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T01:10:43.670 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/remote/smithi116/log 2024-04-26T01:10:43.671 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T01:10:43.999 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/remote/smithi164/log 2024-04-26T01:10:44.000 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T01:10:44.364 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T01:10:44.376 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T01:10:44.376 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T01:10:44.379 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T01:10:44.381 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T01:10:44.383 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T01:10:44.498 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T01:10:44.507 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T01:10:44.507 DEBUG:teuthology.orchestra.run.smithi008:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:10:44.510 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:10:44.512 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:10:44.514 DEBUG:teuthology.orchestra.run.smithi164:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:10:44.522 INFO:teuthology.orchestra.run.smithi008.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:10:44.522 INFO:teuthology.orchestra.run.smithi008.stdout:============================================================================== 2024-04-26T01:10:44.522 INFO:teuthology.orchestra.run.smithi008.stdout:*hv01.front.sepi 67.205.162.81 3 u - 64 177 0.057 -0.405 0.350 2024-04-26T01:10:44.522 INFO:teuthology.orchestra.run.smithi008.stdout:+hv02.front.sepi 63.231.80.2 3 u 65 64 77 0.078 +2.311 0.410 2024-04-26T01:10:44.522 INFO:teuthology.orchestra.run.smithi008.stdout:+hv03.front.sepi 74.6.168.72 3 u 63 64 77 0.080 +1.957 0.409 2024-04-26T01:10:44.522 INFO:teuthology.orchestra.run.smithi008.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:10:44.527 INFO:teuthology.orchestra.run.smithi116.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:10:44.527 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================== 2024-04-26T01:10:44.527 INFO:teuthology.orchestra.run.smithi116.stdout:+hv01.front.sepi 67.205.162.81 3 u 66 64 77 0.070 -1.510 0.702 2024-04-26T01:10:44.527 INFO:teuthology.orchestra.run.smithi116.stdout:+hv02.front.sepi 63.231.80.2 3 u 2 64 177 0.096 +0.829 0.770 2024-04-26T01:10:44.527 INFO:teuthology.orchestra.run.smithi116.stdout:*hv03.front.sepi 74.6.168.72 3 u 64 64 77 0.100 +0.504 0.622 2024-04-26T01:10:44.527 INFO:teuthology.orchestra.run.smithi116.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:10:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:10:44.528 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================== 2024-04-26T01:10:44.528 INFO:teuthology.orchestra.run.smithi012.stdout:+hv01.front.sepi 67.205.162.81 3 u 4 64 177 0.094 +0.021 1.487 2024-04-26T01:10:44.528 INFO:teuthology.orchestra.run.smithi012.stdout:+hv02.front.sepi 63.231.80.2 3 u 6 64 177 0.071 +2.289 1.439 2024-04-26T01:10:44.528 INFO:teuthology.orchestra.run.smithi012.stdout:*hv03.front.sepi 74.6.168.72 3 u 5 64 177 0.099 +0.324 1.366 2024-04-26T01:10:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:10:44.529 INFO:teuthology.orchestra.run.smithi164.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:10:44.529 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================== 2024-04-26T01:10:44.529 INFO:teuthology.orchestra.run.smithi164.stdout:+hv01.front.sepi 67.205.162.81 3 u - 64 177 0.089 -1.979 0.777 2024-04-26T01:10:44.529 INFO:teuthology.orchestra.run.smithi164.stdout:+hv02.front.sepi 63.231.80.2 3 u 5 64 177 0.054 -1.085 1.194 2024-04-26T01:10:44.529 INFO:teuthology.orchestra.run.smithi164.stdout:*hv03.front.sepi 74.6.168.72 3 u 6 64 177 0.079 -1.419 1.186 2024-04-26T01:10:44.529 INFO:teuthology.orchestra.run.smithi164.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:10:44.529 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T01:10:44.542 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T01:10:44.542 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T01:10:44.570 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T01:10:44.606 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T01:10:44.640 INFO:teuthology.task.internal:Duration was 852.869548 seconds 2024-04-26T01:10:44.640 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T01:10:44.673 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T01:10:44.673 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T01:10:44.676 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T01:10:44.678 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T01:10:44.680 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T01:10:44.720 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T01:10:44.720 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi008.front.sepia.ceph.com 2024-04-26T01:10:44.721 DEBUG:teuthology.orchestra.run.smithi008:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T01:10:44.765 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2024-04-26T01:10:44.766 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T01:10:44.776 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-04-26T01:10:44.776 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T01:10:44.786 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi164.front.sepia.ceph.com 2024-04-26T01:10:44.786 DEBUG:teuthology.orchestra.run.smithi164:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T01:10:44.796 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T01:10:44.797 DEBUG:teuthology.orchestra.run.smithi008:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T01:10:44.809 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T01:10:44.819 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T01:10:44.831 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T01:10:44.862 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T01:10:44.862 DEBUG:teuthology.orchestra.run.smithi008:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T01:10:44.877 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T01:10:44.887 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T01:10:44.903 DEBUG:teuthology.orchestra.run.smithi164:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T01:10:44.979 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T01:10:44.989 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T01:10:44.989 DEBUG:teuthology.orchestra.run.smithi008:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T01:10:45.001 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T01:10:45.016 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T01:10:45.029 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T01:10:45.041 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T01:10:45.054 DEBUG:teuthology.orchestra.run.smithi008:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:10:45.056 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:10:45.059 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:10:45.066 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern = core 2024-04-26T01:10:45.069 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2024-04-26T01:10:45.075 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:10:45.083 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-04-26T01:10:45.096 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = core 2024-04-26T01:10:45.150 DEBUG:teuthology.orchestra.run.smithi008:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:10:45.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:10:45.155 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:10:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:10:45.160 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:10:45.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:10:45.165 DEBUG:teuthology.orchestra.run.smithi164:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:10:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:10:45.197 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T01:10:45.211 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T01:10:45.211 DEBUG:teuthology.misc:Transferring archived files from smithi008:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/remote/smithi008 2024-04-26T01:10:45.212 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T01:10:45.234 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/remote/smithi012 2024-04-26T01:10:45.235 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T01:10:45.255 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/remote/smithi116 2024-04-26T01:10:45.256 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T01:10:45.276 DEBUG:teuthology.misc:Transferring archived files from smithi164:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673968/remote/smithi164 2024-04-26T01:10:45.277 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T01:10:45.297 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T01:10:45.297 DEBUG:teuthology.orchestra.run.smithi008:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T01:10:45.300 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T01:10:45.302 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T01:10:45.319 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T01:10:45.346 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T01:10:45.360 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T01:10:45.360 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T01:10:45.427 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T01:10:45.427 DEBUG:teuthology.orchestra.run.smithi008:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T01:10:45.430 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T01:10:45.432 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T01:10:45.434 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T01:10:45.435 INFO:teuthology.orchestra.run.smithi008.stdout: 658469 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 01:10 /home/ubuntu/cephtest 2024-04-26T01:10:45.436 INFO:teuthology.orchestra.run.smithi012.stdout: 658643 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 01:10 /home/ubuntu/cephtest 2024-04-26T01:10:45.438 INFO:teuthology.orchestra.run.smithi116.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 01:10 /home/ubuntu/cephtest 2024-04-26T01:10:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: 658778 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 01:10 /home/ubuntu/cephtest 2024-04-26T01:10:45.442 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T01:10:45.473 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T01:10:45.541 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} duration: 852.8695480823517 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-26T01:10:45.541 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T01:10:45.623 INFO:teuthology.run:pass